Authentication device, biological information acquisition device, and electronic device

Abstract

A small-scale and highly accurate biometric information acquisition device is provided. The acquired photoelectric pulse wave is converted into multiple frequency components using a Fast Fourier Transform (FFT). A portion of this data is quantized and input into a neural network (NN) for personal identification or emotion recognition. By quantizing the data to a low number of bits, a small-scale NN can be used, enabling high-speed decision-making. Furthermore, accuracy is improved by quantizing the data to reflect the symbolic representation. Due to the small size of the NN, biometric information acquisition and processing can be performed quickly and securely without the need for high-performance workstations in the cloud.

Description

Technical Field

[0001] One aspect of the present invention relates to a bio-information acquisition device and an electronic device including the bio-information acquisition device.

[0002] Note that one aspect of the present invention is not limited to the technical fields described above. Examples of technical fields encompassing one aspect of the present invention disclosed in this specification include semiconductor devices, display devices, light-emitting devices, energy storage devices, memory devices, electronic devices, lighting devices, input devices, input / output devices, methods for driving these devices, and methods for manufacturing these devices. A semiconductor device refers to any device capable of operating by utilizing the characteristics of semiconductors. Background Technology

[0003] In the information society, to prevent unauthorized access, biometric authentication, a stronger authentication method than password authentication, is being promoted. Biometric authentication is an authentication method that uses personally unique biometric information. For example, most methods primarily use image data, such as fingerprint authentication, palm print authentication, iris authentication, and facial recognition.

[0004] Biometric authentication using image data identifies an individual by analyzing the image data obtained during authentication and determining whether it matches pre-registered image data. For example, Patent Document 1 discloses an example of using a camera device equipped with a light source for personal authentication.

[0005] Artificial intelligence can be used to infer whether the image data matches the aforementioned image data. For example, Non-Patent Document 1 reports an example of an AI accelerator that includes multiple neural networks and is capable of performing different inferences in parallel.

[0006] In addition, there is research on recognizing human emotions. If we could understand human emotions, we could provide services tailored to those emotions, store emotional data, and use it for healthcare, among other things. Furthermore, by applying external stimuli in ways that alleviate negative emotions, we could prevent crimes or accidents.

[0007] Human emotions are easily expressed through facial expressions, but people can conceal outward changes, making it difficult to understand a person's true emotions solely from their appearance. One bodily change that is difficult for humans to consciously control is the pulse wave. Non-patent documents 2 and 3 report on emotion recognition using pulse waves. [Preliminary Technology Documents] [Patent Literature]

[0008] [Patent Document 1] WO2020 / 049398 [Non-patent literature]

[0009] [Non-patent document 1] Y. Yakubo et al., "Crystalline Oxide Semiconductor-based3D Bank Memory System for Endpoint Artificial Intelligence with MultipleNeural Networks Facilitating Context Switching and Power Gating," ISSCC, 2023. [Non-patent document 2] Lee MS, Lee YK, Pae DS, Lim MT, Kim DW, Kang TK. “FastEmotion Recognition Based on Single Pulse PPG Signal with ConvolutionalNeural Network. Applied Sciences. 2019; 9(16): 3355. [Non-patent document 3] S. -W. Wang and S. -N. Yu, "Emotion Recognition Based on Photoplethysmography Using ResNet and BiLSTM Networks," 2021 International Conference on e-Health and Bioengineering (EHB), Iasi, Romania, 2021, pp. 1-4, doi: 10.1109 / EHB52898.2021.9657742. Summary of the Invention The technical problem that the invention aims to solve

[0010] Although biometric authentication using image data is a strong authentication method, it sometimes requires high-resolution camera equipment and large-scale information processing equipment to handle the massive image information.

[0011] Furthermore, while authentication devices are important as a security measure, they are merely auxiliary devices before proceeding to the next step. Additionally, emotion recognition devices require integration with other devices for practical use. That is, in many cases, biometric acquisition devices such as authentication and emotion recognition devices are not used independently but are connected to or integrated into other devices. Therefore, personal identification or emotion recognition requires sufficient discriminative capabilities, but also demands simple, small-scale biometric acquisition devices.

[0012] Furthermore, in biometric authentication using image data, glasses, masks, hats, gloves, or injuries can sometimes hinder the accuracy of information. Moreover, image data can sometimes be inaccurately obtained due to the influence of natural or artificial light. Additionally, as mentioned above, people can conceal their emotions, making it difficult to interpret emotions from appearance.

[0013] Therefore, one objective of this invention is to provide a small-scale and highly accurate bio-information acquisition device. Another objective of this invention is to provide a bio-information acquisition device that easily acquires accurate information. Another objective of this invention is to provide a bio-information acquisition device that can acquire information without being significantly affected by the surrounding environment. Another objective of this invention is to provide a novel bio-information acquisition device. Another objective of this invention is to provide a method of operating the above-described bio-information acquisition device. Another objective of this invention is to provide an electronic device including the bio-information acquisition device. Another objective of this invention is to provide a novel semiconductor device, etc.

[0014] Note that the description of these objectives does not preclude the existence of other objectives. Note that one embodiment of the invention does not necessarily require achieving all of the above objectives. Note that objectives other than those described above can be extracted from the description in the specification, drawings, claims, etc. means of solving technical problems

[0015] One aspect of the present invention relates to a small-scale and highly accurate bioinformatics acquisition device.

[0016] One aspect of the present invention is a bio-information acquisition device having the following functions: acquiring pulse waves in the form of analog data; converting analog data into digital data; performing fast Fourier transform on the digital data; extracting multiple data points from the low-frequency side of the data after the fast Fourier transform; quantizing each of the multiple data points; and performing personal authentication or emotion recognition by inference from the quantized data points.

[0017] The pulse wave described above is preferably a photoelectric pulse wave. Furthermore, the photoelectric pulse wave is preferably obtained at a sampling frequency of 10Hz or higher and 100Hz or lower.

[0018] The number of data extracted from the low-frequency side is preferably 150 or more and 200 or less. Furthermore, this data is preferably data up to 12Hz.

[0019] The quantized data is preferably 5-valued data that reflects the 4-bit representation of the flag.

[0020] In inferences using quantized data, fully connected neural networks are preferred.

[0021] It can identify emotions based on either positive or negative values ​​of comfort or arousal. Additionally, it can identify emotions such as "joy," "anger," "sorrow," and "happiness."

[0022] One aspect of the present invention is a watch-type electronic device comprising a bio-information acquisition device, a watch case, a light sensor module, and a light-receiving window, wherein the light-receiving window is disposed on the surface of the watch case that contacts the arm, and the light sensor module is disposed within the watch case in a manner overlapping the light-receiving window. Furthermore, the bio-information acquisition device may have the function of inferring a person's physical state from pulse waves using a neural network model that has learned the relationship between pulse waves and brain waves. Invention Effects

[0023] One aspect of the present invention can provide a bio-information acquisition device that easily obtains accurate information. Additionally, a bio-information acquisition device can be provided that can acquire information without being significantly affected by the surrounding environment. Furthermore, a small-scale and highly accurate bio-information acquisition device can be provided. Furthermore, a novel bio-information acquisition device can be provided. Furthermore, a method for operating the above-described bio-information acquisition device can be provided. Additionally, a novel semiconductor device and its operating method can be provided.

[0024] Note that the description of these effects does not preclude the existence of other effects. One embodiment of the invention does not necessarily require all of the aforementioned effects. Note that effects other than those described above can be extracted from the description in the specification, drawings, claims, etc. Attached Figure Description

[0025] Figure 1 This is a block diagram illustrating a bioinformatics acquisition device. Figure 2 This is a flowchart illustrating the operation of the bioinformatics acquisition device. Figure 3 A is a diagram showing the photoelectric pulse wave (after digital conversion). Figure 3 B is a graph showing the data after the Fourier transform of the photoelectric pulse wave. Figure 4 A and Figure 4 B is a graph showing the accuracy versus cross-entropy error of the neural network. Figure 5 This is a graph showing the accuracy of each quantization method. Figure 6 This is a diagram illustrating an example of a neural network. Figure 7 This is a diagram illustrating the emotion selection model. Figure 8 A and Figure 8 B is a diagram illustrating the inference results based on emotions. Figure 9 A to Figure 9 D is a diagram illustrating a bioinformatics acquisition device. Figure 9 E and Figure 9 F is a diagram illustrating an electronic device that includes a bio-information acquisition device. Figure 10 A to Figure 10 C is a diagram illustrating an electronic device that includes a bio-information acquisition device. Figure 11 A and Figure 11 B is a diagram illustrating an electronic device that includes a bio-information acquisition device. Detailed Implementation

[0026] The embodiments will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the following description, and those skilled in the art will readily understand that its methods and details can be varied in many ways without departing from the spirit and scope of the invention. Therefore, the present invention should not be construed as limited only to the contents described in the embodiments shown below. Note that in the structure of the invention described below, the same reference numerals are used in different drawings to denote the same parts or parts having the same function, and repeated descriptions are omitted. Note that sometimes the shading of the same constituent elements is appropriately omitted or changed in different drawings.

[0027] In this specification, "connection" includes, for example, "electrical connection." Note that sometimes "electrical connection" is used to describe the connection relationship of circuit elements as an object. Furthermore, "electrical connection" includes both "direct connection" and "indirect connection." "A and B directly connected" means that A and B are connected without any circuit elements (e.g., transistors, switches, etc. Note that wiring is not a circuit element). On the other hand, "A and B indirectly connected" means that A and B are connected through more than one circuit element.

[0028] For example, assuming a circuit including A and B is operating, if there are opportunities during circuit operation where electrical signals are exchanged or potentials interact between A and B, such a circuit can be defined as "A and B are indirectly connected". Furthermore, even if there are times during circuit operation where no electrical signals are exchanged or potentials interact between A and B, the opportunity where electrical signals are exchanged or potentials interact between A and B can still be defined as "A and B are indirectly connected".

[0029] An example of "A and B being indirectly connected" is a case where A and B are connected through the source and drain of more than one transistor. On the other hand, an example where "A and B being indirectly connected" cannot be said is a case where there is an insulator in the path from A to B. Specifically, this includes cases where a capacitor is connected between A and B, and cases where there is a gate insulating film of a transistor between A and B. Therefore, it cannot be said that "the gate (A) of a transistor is indirectly connected to the source or drain (B) of the transistor."

[0030] As another example where it cannot be said that "A and B are indirectly connected", there is the following situation: multiple transistors are connected through the source and drain along the path from A to B, and a fixed potential V is supplied from the power supply, GND, etc. to the nodes between the transistors and other transistors.

[0031] (Implementation Method 1) In this embodiment, a biometric information acquisition device, as one aspect of the present invention, is described using an authentication device that utilizes pulse waves.

[0032] One aspect of the present invention is a small-scale and highly accurate authentication device. This authentication device utilizes biometric pulse waves. Pulse waves are waveforms that show changes in arterial pressure, and information can be obtained with minimal influence from the surrounding environment. Pulse wave or equivalent information can be obtained from electrocardiograms, photoelectric pulse waves, blood pressure, heart sounds, etc. In particular, photoelectric pulse wave information can be obtained using small optical sensors, thus enabling the authentication device to be constructed at low cost.

[0033] In one aspect of the invention, the acquired photoelectric pulse wave is converted into multiple frequency components using a Fast Fourier Transform (FFT). A portion of this data is then quantized and input into a neural network (NN) for personal identification. By quantizing the data into a low number of bits for input, a small-scale NN with fewer input and intermediate layers can be used, enabling high-speed decision-making. Furthermore, accuracy can be improved by quantizing the data to reflect the identifier representation.

[0034] In one aspect of this invention, learning and inference can be performed at high speed due to the use of a small-scale neural network (NN). Furthermore, because the NN is small, computation can be performed using microcomputers, personal computers, tablets, or smartphones, without the need for the high-performance workstations required for existing large-scale NNs. That is, inference processing can be performed without data communication via the Internet, thereby enabling high-speed and secure authentication processing.

[0035] Figure 1This is a block diagram illustrating one aspect of an authentication device according to the present invention. The authentication device 130 may include a sensor 101, a microcomputer 120, and a display unit 110. Note that... Figure 1 The block diagram shown can also be used in the emotion recognition device 140 described later. Therefore, in Figure 1 In this context, the authentication device 130 and the emotion recognition device 140 are collectively referred to as 130 (140).

[0036] The authentication device 130 has the function of determining the biometric information obtained by the sensor 101 by a microcomputer 120 with NN function and displaying the authentication result on the display unit 110. In addition, the microcomputer 120 is not limited to a single-chip microcomputer, but can also be a single-board microcomputer.

[0037] Furthermore, the authentication device 130 is not limited to a standalone device; it can also be part of a device A with other functions. Alternatively, the authentication device 130 can be integrated into device A. In other words, the availability of device A's functions can be determined based on the authentication result of the authentication device 130.

[0038] Furthermore, the components included in the authentication device 130 can also be shared within device A. For example, if device A is an information terminal such as a personal computer, tablet computer, or smartphone, since they include components equivalent to those in the microcomputer 120, device A can share some or all of the components included in the authentication device 130.

[0039] For example, the camera included in a smartphone can be used as sensor 101, the input / output unit, arithmetic unit, and storage unit included in a smartphone can be used as components of a microcomputer 120, and the display unit of a smartphone can be used as display unit 110, etc.

[0040] In one aspect of the invention, a pulse wave is used as the bio-information acquired by sensor 101. Pulse waves can be obtained from devices that measure electrocardiograms, photoelectric pulse waves, blood pressure, or heart sounds, but devices for measuring electrocardiograms and blood pressure are relatively large, and heart sounds are easily affected by environmental noise. Therefore, photoelectric pulse waves are preferred as a method for acquiring pulse waves. Photoelectric pulse waves can be acquired using a small optical sensor in the form of analog data.

[0041] Note that although photoelectric pulse waves are easily affected by ambient light, accurate data can be obtained by placing a light sensor and a light source in the dark. For example, when obtaining data from the fingertip, a light sensor can be placed inside a box-shaped, bag-shaped, or strip-shaped light-shielding object covering the fingertip. Alternatively, data can be obtained in bright light by using a light sensor to obtain the difference between the light and ambient light.

[0042] Because oxyhemoglobin in the blood absorbs light G at wavelengths of 550 nm and around, the amount of light G absorbed changes with the volume of the blood vessels. Therefore, by obtaining the reflected or transmitted light of light G illuminating a living organism, the pulse wave that changes with the heart's pulsation can be obtained.

[0043] As the optical sensor for detecting light G, examples include photodiodes or phototransistors. Furthermore, as the light source, LED elements that emit green light are preferred. Therefore, an optical sensor module that includes a light source is preferably used as the sensor 101.

[0044] The microcomputer 120 includes an input unit 102, an arithmetic unit 103, an arithmetic unit 104, a storage unit 105, a storage unit 106, an ADC (analog-to-digital converter) 107, a display driver 108, and an output unit 109. Furthermore, the above-mentioned components are connected via a bus 115. Alternatively, the microcomputer 120 may include components other than those described above. Alternatively, some of the above-mentioned components may be omitted. Alternatively, a structure combining several of the above-mentioned components may be used.

[0045] The input unit 102 has the function of an interface for inputting information from the outside. Note that the input information includes analog data of biological information input from sensor 101, learned neural network models, etc. In addition, biological information data obtained from external device 112 can also be input to the input unit 102. This data is not limited to photoelectric pulse waves, but can also be data such as electrocardiogram, blood pressure, heart sounds, or brain waves. Biometric authentication can also be performed by using this data instead of pulse waves. Furthermore, the data input from the outside is not limited to analog data, but can also be digital data.

[0046] The arithmetic unit 103 may include, for example, a central processing unit (CPU). The arithmetic unit 103 may have the functions of each component such as a control input unit 102, an arithmetic unit 103, an arithmetic unit 104, a storage unit 105, a storage unit 106, an ADC 107, a display driver 108, and an output unit 109.

[0047] Signals are transmitted between the arithmetic unit 103 and each component via the bus 115. The arithmetic unit 103 has the functions of processing signals input from each component connected via the bus 115 and generating signals output to each component, and can control each component connected to the bus 115 in an overall manner.

[0048] The arithmetic unit 103 performs various data processing and program control by interpreting and executing instructions from various programs by the CPU. The programs executed by the arithmetic unit 103 can be stored in the memory area provided in the arithmetic unit 103, or in the memory unit 105 or memory unit 106 described later.

[0049] As the computing unit 104, a computing device specifically designed for parallel computing compared to a CPU is preferably used. For example, computing devices comprising multiple (tens to hundreds) cores capable of parallel processing, such as GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), and NPUs (Neural Processing Units), are preferred. These can be used as AI accelerators and can perform neural network (NN) operations at high speed. Furthermore, as the NN, a fully connected NN or a convolutional NN can be used.

[0050] Alternatively, the arithmetic unit 104 can be implemented using PLDs (Programmable Logic Devices) such as FPGAs (Field Programmable Gate Arrays) or FPAAs (Field Programmable Analog Arrays). Furthermore, a dedicated structure for an AI accelerator implemented using ASICs (Application Specific Integrated Circuits) can also be employed. Note that in Figure 1 In this context, the arithmetic unit 103 and the arithmetic unit 104 are different constituent elements, but they can also be integrated constituent elements.

[0051] As the storage unit 105, a storage device including storage elements such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory) can be used. Although these storage elements are volatile, they can achieve high-speed access, so the storage unit 105 can be used as an auxiliary storage area for the arithmetic unit 103 and the arithmetic unit 104. In addition, these storage elements can also be used as a storage area for data input from the sensor 101, a storage area for data converted from analog to digital, a storage area for output data after processing, or a temporary storage area for programs read from the storage unit 106.

[0052] As the storage unit 106, for example, storage devices including non-volatile storage elements such as flash memory, MRAM (Magnetoresistive Random Access Memory), PRAM (Phase Change RAM), ReRAM (Resistive RAM), and FeRAM (Ferroelectric RAM) can be used. Alternatively, mask ROM, OTPROM (One Time Programmable Read Only Memory), and EPROM (Erasable Programmable Read Only Memory) can be used. Furthermore, recording media drives such as hard disk drives (HDDs) and solid-state drives (SSDs) can also be used.

[0053] Although they are non-volatile, their access speed is slow. Therefore, it is preferable to store information that is not frequently accessed by the arithmetic unit 103 and the arithmetic unit 104 and does not need to be rewritten or is rewritten only occasionally. For example, BIOS (Basic Input / Output System), firmware, operating system, application programs, programs related to the operation of the authentication device 130, learning information of the NN, etc. can be stored.

[0054] At least a portion of the constituent elements included in the aforementioned arithmetic units 103, 104 and storage units 105, 106 preferably include transistors (hereinafter, OS transistors) using oxide semiconductors as semiconductor layers. Because the off-state current of an OS transistor is extremely small, it is preferable to use the OS transistor as a switch to retain the charge (data) flowing into a capacitor used as a storage element. By using the OS transistor as this switch, data can be retained for a long period. By applying this characteristic to at least one of the registers and cache memories included in the arithmetic unit, the arithmetic unit can be operated only when necessary, while previously processed information can be stored in the storage element and the arithmetic unit can be turned off in other situations. In other words, normally offcomputing can be implemented, thereby achieving low power consumption in the microcomputer 120.

[0055] ADC107 is an analog-to-digital converter circuit that converts analog data input to input unit 102 into digital data. For example, it can convert analog data of biological information acquired by sensor 101 into 10-bit digital data. Alternatively, arithmetic unit 103 or arithmetic unit 104 may also have the function of ADC107. Furthermore, sensor 101 may also have the function of outputting digital data. In this case, ADC107 can be omitted.

[0056] The display driver 108 is a drive circuit that generates image data for displaying authentication results and outputs the image data and drive signals to the display unit 110. Note that the type of drive circuit varies depending on the type of display unit 110. Furthermore, although in Figure 1 The example shown is the output of the authentication result as displayed on the display unit 110, but the authentication result can also be output as sound or vibration.

[0057] The output unit 109 has an interface for outputting data processed by the authentication device 130 to an external source. For example, the learning and inference of the neural network can be performed within the microcomputer 120, but data acquired by the sensor 101 can also be output to an external source via the output unit 109 and performed within the AI ​​workstation (AI-WS) 111 in the cloud. By performing some data processing within the microcomputer 120 and then sending the data to the AI ​​workstation 111, the power consumption for data transmission and the load on the AI ​​workstation 111 can be reduced.

[0058] The learning model or inference results of the neural network generated by the AI ​​workstation 111 can be input to the microcomputer 120 via the input unit 102. Note that the external neural network learning and inference device is not limited to the AI ​​workstation 111, but can also be a personal computer, tablet terminal, smartphone, etc.

[0059] Additionally, information regarding the authentication result can be output from the output unit 109 to the external device 113. The external device 113 can then grant access rights to the user based on the authentication result.

[0060] Note that, although in Figure 1The example shown is an input unit 102 and an output unit 109 respectively, but they can also be combined into a single input / output unit. Furthermore, the input unit 102, output unit 109, or the aforementioned input / output unit can also have wireless communication capabilities. When performing wireless communication, the following communication protocols or technologies can be used: communication standards such as LTE (Long Term Evolution); or specifications standardized by the IEEE (Institute of Electrical and Electronics Engineers) such as Wi-Fi (Wireless Fidelity) and Bluetooth.

[0061] Next, refer to Figure 2 The flowchart shown illustrates an example of the operation of the authentication device 130. Note that the right side of the flowchart shows the main steps corresponding to each step. Figure 1 The elements.

[0062] First, in step S1, sensor 101 is used to acquire pulse wave information (analog data). Here, it is necessary to accurately obtain the shape of the pulse wave, but for efficient subsequent calculations, the amount of data is preferably small.

[0063] In other words, preferably, dozens of numerical data points are obtained from each pulse wave (corresponding to the waveform of one beat). Since the normal human pulse rate is 60 to 100 beats per minute, the sampling frequency is above 10 Hz and below 100 Hz, preferably above 20 Hz and below 80 Hz, and more preferably above 30 Hz and below 60 Hz.

[0064] Preferably, the sampling time is longer during the learning phase of the neural network (NN) and shorter during authentication. For example, it is preferable to set the sampling time to approximately 15 seconds during learning to acquire multiple pulse waves, and to set the sampling time to more than 1 second and less than 3 seconds during authentication to acquire at least one pulse wave. To further improve authentication accuracy, it is preferable to set the sampling time to more than 2 seconds and less than 6 seconds during authentication to acquire two or three pulse waves.

[0065] Next, in step S2, the pulse wave (analog data) is converted into digital data using the ADC107. Figure 3 A is an example of a pulse wave represented by a numerically converted value. Note that... Figure 3 The data shown in A is obtained using a sampling frequency of 64Hz.

[0066] Next, in step S3, the arithmetic unit 103 performs a fast Fourier transform on the digital data of the pulse wave to decompose it into frequency components. Figure 3 B is Figure 3Figure A shows the Fast Fourier Transform (FFT) result of the pulse wave, with the amplitude spectrum normalized. Note that the method shown here utilizes the FFT, but the Discrete Cosine Transform (DCT) can also be used.

[0067] Next, in step S4, multiple consecutive data points from the low-frequency side are extracted from the result of the Fast Fourier Transform. For example... Figure 3 As shown in Figure B, almost no distinct frequency components are extracted on the high-frequency side, which is higher than approximately 12 Hz. Therefore, the feature quantity can be considered to be located on the low-frequency side, which is lower than approximately 12 Hz, and the high-frequency side can be ignored. Furthermore, limiting the data range is effective in miniaturizing the neural network.

[0068] Here, we explain the results of calculating and comparing the accuracy of NN inferences as the amount of data changes in order to determine the effective data range. Note that a fully connected NN model is used in the calculation, where the input data is 32-bit floating-point numbers, the intermediate layers are 128×3 layers (activation function = Tanh), and the output layer is 10 (activation function = softmax).

[0069] Additionally, during step S1, pulse wave information for four individuals is obtained, with 15.6 seconds (64Hz, 1000 points) × 100 data points collected from each individual. This data is then processed through steps S2 to S4, with 80% used for learning and 20% for inference. The accuracy rate is calculated by comparing each individual-specific number added to the output layer during learning with the number in the output layer that represents the maximum value during inference, thus identifying the individual.

[0070] Figure 4 A is a graph showing the relationship between the amount of data (the number of consecutive data points drawn from the low-frequency side) and the accuracy at inference time, as well as the relationship between the amount of data and the cross-entropy error at both the learning and inference times. The cross-entropy error at the learning time is small even with a small amount of data, but the cross-entropy error at the inference time tends to continue to decrease until the amount of data exceeds 150. This region represents overlearning caused by a small amount of data.

[0071] As mentioned above, the number of data points is preferably 150 or more, but from the viewpoint of reducing the size of the neural network, a smaller number of data points is preferable. Therefore, the number of data points is preferably 150 or more but close to 150; for example, an effective range of data points is approximately 150 or more but less than 200. Furthermore, a data point of 150 can be considered a point where the accuracy begins to show a saturation tendency. Therefore, to obtain more accurate inference results, the following explanation uses an optimal data point of 175.

[0072] Figure 4 B is to Figure 4The horizontal axis of A is replaced with the maximum frequency. The maximum frequency corresponding to data point 175 is between 11Hz and 12Hz, therefore it can be said that the data up to approximately 12Hz is valid. From this result, we can see that... Figure 3 The explicit data represented by the Fast Fourier Transform results shown in B are almost all used effectively.

[0073] Next, in Figure 2 In step S5, the data extracted in step S4 (175 data points as an example) is quantized. The data extracted in step S4 is a 32-bit floating-point number, so a large-scale neural network is needed for direct computation. Therefore, the data is quantized to a low number of bits to handle small-scale neural networks.

[0074] There are many quantization methods, among which the inventors have tried the following: a method of using 32-bit floating-point numbers of the original data as a reference for operation; method A of dividing the original data into 4 bits; method B of dividing the original data into 4 bits after performing a logarithmic transformation; and method C of dividing the original data using several thresholds and representing it with 4 bits of data reflecting the flag representation.

[0075] Figure 5 This is a graph showing the accuracy of inferences made using the methods described above. Aside from the reference, the most accurate method is method C, which uses data reflecting the symbol representation, achieving over 90% accuracy, close to the 95% of the reference.

[0076] The details of method C are as follows. First, the 32-bit floating-point number used as the original data is normalized to 0 to 255, and the thresholds for the 0th bit, the 1st bit, the 2nd bit, and the 3rd bit are set respectively. When the value of the normalized original data (hereinafter, data S) is above each threshold, the flag (1) is set.

[0077] Specifically, when the value of data S is less than the threshold of bit 0, it is represented as

[0000] . When the value of data S is above the threshold of bit 0 and less than the threshold of bit 1, a flag is set in bit 0 and represented as

[0001] . When the value of data S is above the threshold of bit 1 and less than the threshold of bit 2, flags are set in bits 0 and 1 and represented as

[0011] . When the value of data S is above the threshold of bit 2 and less than the threshold of bit 3, flags are set in bits 0 to 2 and represented as

[0111] . When the value of data S is above the threshold of bit 3, flags are set in bits 0 to 3 and represented as

[1111] . In other words, data S is classified into 5 values ​​represented by 4 bits. This representation is called flag representation.

[0078] Table 1 shows the results of calculations using the fully connected neural network model described above, with multiple different thresholds applied to the 175 data points S extracted in step S4 according to the aforementioned rules. It can be seen that in experiments No.1 to No.13, the cross-entropy error and accuracy vary depending on the threshold conditions. Among these, considering learning, the threshold settings for No.7 or No.8, which have smaller cross-entropy error values ​​during inference, can be considered appropriate.

[0079] [Table 1]

[0080] Note that the thresholds in Table 1 are values ​​experimentally set by the inventors, but thresholds more suitable for NN learning can be set using cross-entropy error and accuracy as criteria. Furthermore, although the example shown above classifies data S into 5 values ​​represented by 4 bits, it is not limited to this. For example, it can also be classified into 4 values ​​represented by 3 bits, 6 values ​​represented by 5 bits, etc.

[0081] Next, in Figure 2 In step S6, the quantized data from step S5 is input into the neural network (NN) included in the computation unit 104 for computation. The NN can be a fully connected NN or a convolutional NN. In steps S4 and S5, each of the 175 data points is quantized into 4 bits, therefore the NN only needs to be input with a minimum of 700 1-bit data.

[0082] Figure 6 The diagram shows a fully connected neural network (NN) with an input layer of 784, an intermediate layer of 128×3, and an output layer of 10, as shown in Non-Patent Document 1. By quantizing pulse wave data in a manner that is feasible with this small-scale NN, high-speed inference can be performed.

[0083] In step S7, if the above calculation is performed during learning, the process returns to step S1, and steps S2 to S6 are repeated using different data. If the above calculation is performed during inference, the process proceeds to step S8, where personal identification is performed and the authentication result is displayed on display unit 110. Furthermore, some or all of the learning and inference processes can be performed in the AI ​​workstation 111, etc.

[0084] As described above, by using pulse waves as bio-information and appropriately quantizing the data, personal identification can be performed even using a small-scale neural network. That is, by using one aspect of the present invention, a small-scale and highly accurate authentication device can be formed.

[0085] At least a portion of this embodiment can be implemented in combination with other embodiments described in this specification.

[0086] (Implementation Method 2) In this embodiment, a pulse wave-based emotion recognition device is described as one aspect of the present invention for acquiring biological information.

[0087] Note that the emotion recognition device described in this embodiment can be composed of the same components as the authentication device 130 described in Embodiment 1, and the structure of the device used as the NN and the operation of the authentication device 130 can also be substantially the same. Therefore, regarding the structure of the emotion recognition device described in this embodiment and the device used as the NN, detailed descriptions can be omitted by referring to the description of the structure of the authentication device 130 described above.

[0088] One aspect of the present invention is a small-scale and highly accurate emotion recognition device. This emotion recognition device utilizes bio-information pulse waves. In one aspect of the present invention, because a small-scale neural network (NN) is used, learning and inference can be performed at high speed. Furthermore, because the NN is small in scale, computation can be performed without the high-performance workstations required for existing large-scale NNs. That is, inference processing can be performed without data communication via the Internet, thereby enabling high-speed and secure recognition of human emotions.

[0089] By identifying human emotions, for example, services corresponding to emotions can be provided, emotion shift data can be used for healthcare, or crime or accidents can be prevented by externally applying stimuli in a way that alleviates negative emotions.

[0090] Similar to authentication device 130, emotion recognition device can employ... Figure 1 The block diagram shown illustrates the structure of the emotion recognition device 140. The emotion recognition device 140 may include a sensor 101, a microcomputer 120, and a display unit 110.

[0091] The authentication device 130 shown in Embodiment 1 is used for user self-authentication; therefore, a contact-type optical sensor module with a clearly defined method of operation is preferably used as the sensor 101. On the other hand, the emotion recognition device is used not only to recognize the user's emotions at the time the user expects, but also to recognize the user's emotions at unexpected times or to recognize the emotions of people other than the user. Therefore, not only a contact-type sensor can be used as the sensor 101, but also a camera module or similar device capable of acquiring information in a non-contact manner.

[0092] Next, refer to Figure 2 The flowchart shown illustrates an example of how the emotion recognition device 140 works.

[0093] First, pulse wave information (analog data) is obtained in step S1, and then the analog data is converted into digital data in step S2. For details on the appropriate conditions for obtaining the pulse wave, please refer to the description of the operation of the authentication device 130.

[0094] Emotion recognition devices require learning data corresponding to each of a person's various emotions. In this embodiment, two test subjects were shown dynamic images (a total of 14) lasting up to approximately 5 minutes each, representing any of the predicted emotions of joy, anger, sorrow, and happiness, and their pulse waves during the viewing of the dynamic images were obtained.

[0095] In Russell's circular model of emotion, a person's emotions can be represented by a combination of comfort (unpleasant-pleasant) and arousal (calm-excited). Figure 7 This is an emotion selection model based on Russell's emotional ring model. The X-axis represents the emotional valence, where negative (-) areas represent unpleasant emotions and positive (+) areas represent pleasant emotions. The Y-axis represents the emotional valence, where negative (-) areas represent unarousal emotions and positive (+) areas represent arousal emotions.

[0096] Figure 7 The model has four quadrants. Quadrant 1 (Q1) represents emotions with positive comfort and positive arousal, defined as "joy" in the four-emotion spectrum. Quadrant 2 (Q2) represents emotions with negative comfort and positive arousal, defined as "anger" in the four-emotion spectrum. Quadrant 3 (Q3) represents emotions with negative comfort and negative arousal, defined as "sorrow" in the four-emotion spectrum. Quadrant 4 (Q4) represents emotions with positive comfort and negative arousal, defined as "joy" in the four-emotion spectrum. Note that the value 0 is not included in any quadrant, so positive values ​​are ≥0 and negative values ​​are <0.

[0097] After viewing the animated images, the emotional intensity of each image was rated. In the rating, comfort (X) and arousal (Y) were both divided into nine levels from -4 to +4. The pulse waves obtained while viewing each animated image were labeled as follows: comfort level was labeled as positive or negative (left or right in the model); arousal level was labeled as positive or negative (up or down in the model); and emotion was labeled as belonging to quadrant Q1 to quadrant Q4.

[0098] For example, when focusing on quadrants Q1 through Q4, if the comfort level is +2 and the arousal level is +3 when viewing a moving image, the pulse wave data is labeled as "joy" in quadrant Q1. Conversely, if the comfort level is -1 and the arousal level is -4 when viewing a moving image, the pulse wave data is labeled as "sorrow" in quadrant Q3. These labeled pulse wave data can be used as learning data.

[0099] Note that pulse waves change not only from viewing moving images but are also affected by other factors. For example, pulse waves can change due to external stimuli such as sound, physical contact, or physical activity. Alternatively, noise or defects may sometimes be present in the device itself that acquires the pulse wave. Such pulse wave data is noise for the learning and inference of the neural network and is therefore preferably removed.

[0100] For example, in the obtained pulse wave data (refer to...) Figure 3 A) If the difference between the maximum and minimum values ​​in a pulse wave is greater than value A or less than value B, the data can be deleted. Values ​​A and B can be freely set within the range that can be determined as abnormal pulse waves. Note that noise removal can also be performed after the Fourier transform. Note that as long as a stable pulse wave can be obtained, noise removal is not necessary.

[0101] Next, in step S3, the digital data of the pulse wave is decomposed into frequency components by performing a Fast Fourier Transform. Preferably, digital data of the pulse wave over several seconds is acquired, using data acquired within a timeframe sufficient to capture one or two pulse waves. In this embodiment, 256 data points acquired at 64Hz over 4 seconds are used.

[0102] Next, in step S4, multiple consecutive data points from the low-frequency side are extracted from the result of the Fast Fourier Transform. Here, 175 data points (data points greater than 0 Hz and up to about 12 Hz) are used, similar to the authentication device 130 described in Embodiment 1.

[0103] Next, in step S5, the data extracted in step S4 is quantized. Here, the 32-bit floating-point number of the original data is normalized to 0 to 15, and the 175 original data (data S) are converted into 4-bit data (

[0000] ,

[0001] ,

[0011] ,

[0111] ,

[1111] ) reflecting the 5 values ​​of the flag representation by providing several thresholds, similar to the quantization in the usage mode C of the authentication device 130 described in Embodiment 1.

[0104] Here, the results of searching for an appropriate threshold using the same fully connected neural network as authentication device 130 are explained. Of the data processed in steps S1 to S4 above, 80% is used for learning and 20% for inference.

[0105] The accuracy of inference is calculated by comparing the number corresponding to each additional label applied to the output layer during learning with the number of the output layer that takes the maximum value during inference. Here, the label refers to the label corresponding to each label applied to the pulse wave obtained when viewing the dynamic image, and each label indicates which of the positive and negative values ​​of comfort level, arousal level, or which of the four quadrants Q1 to Q4 it belongs to.

[0106] In the threshold search, firstly, for each of comfort and arousal levels, multiple thresholds are calculated for the 175 data points S extracted in step S4 according to the above rules. The thresholds used for the search are as follows: the threshold for the 0th position is 0.1 to 0.5 (five conditions for Δ0.1), the threshold for the 1st position is 1.5 to 1.9 (five conditions for Δ0.1), the threshold for the 2nd position is 2.3 to 2.7 (five conditions for Δ0.1), and the threshold for the 3rd position is 3 to 5 (three conditions for Δ1), for a total of 375.

[0107] Table 2 shows the threshold search results suitable for inferences based solely on comfort. Table 3 shows the threshold search results suitable for inferences based solely on arousal. Tables 2 and 3 respectively show the top ten conditions with the highest accuracy from a total of 375 search results. Accuracy represents the integration of the label attached to the pulse wave while viewing moving images with the label inferred by the neural network.

[0108] [Table 2]

[0109] [Table 3]

[0110] Table 2 shows that when inferring only comfort level, the threshold for No. 1 with the highest accuracy (3rd position: 4, 2nd position: 2.7, 1st position: 1.5, 0th position: 0.1) is appropriate. Table 3 shows that when inferring only arousal level, although No. 1 and No. 2 have the same accuracy, the threshold for No. 1 with the smaller cross-entropy error (3rd position: 3, 2nd position: 2.4, 1st position: 1.6, 0th position: 0.3) is appropriate.

[0111] Here, considering the final inference of emotions from the first quadrant Q1 to the fourth quadrant Q4, the optimal state is that the threshold of No.1 in Table 2 is consistent with the threshold of No.1 in Table 3. These thresholds can be set as the thresholds used to infer emotions from the first quadrant Q1 to the fourth quadrant Q4.

[0112] However, as mentioned above, the threshold suitable for comfort inference and the threshold suitable for arousal inference are not necessarily the same. In this case, it is preferable to set the threshold based on the conditions with the most identical thresholds among the conditions with high accuracy.

[0113] Specifically, the values ​​of the first to third positions of No.1 in Table 2 and No.2 in Table 3 are the same, and their accuracy is high. Therefore, it can be said that the appropriate thresholds for the first, second, and third positions are 1.5, 2.7, and 4, respectively.

[0114] Since the threshold for the remaining 0th position differs between No.1 in Table 2 and No.2 in Table 3, the threshold search can be performed again by fixing the thresholds for the 1st to 3rd positions to the above values ​​and changing the threshold for the 0th position based on the emotions in the first quadrant Q1 to the fourth quadrant Q4.

[0115] The thresholds used for the search are: 4 for the 3rd position, 2.7 for the 2nd position, 1.5 for the 1st position, and 0.1 to 0.4 for the 0th position (the four conditions of Δ0.1), for a total of 4.

[0116] Table 4 shows the threshold search results suitable for inferring emotions in quadrants 1 (Q1) to 4 (Q4). As shown in Table 4, the threshold condition No. 1 (3rd position: 4, 2nd position: 2.7, 1st position: 1.5, 0th position: 0.4) has the highest accuracy. The threshold can be determined by using this threshold search method.

[0117] [Table 4]

[0118] After step S6, please refer to the description of the authentication device 130.

[0119] The neural network was trained using the thresholds searched above and the acquired labeled data to validate the inferences about emotions. The inferred emotions were positive and negative values ​​for comfort and arousal, emotions in quadrant Q2 (equivalent to "anger"), and emotions in quadrant Q4 (equivalent to "happiness"). Note that the training objects for "anger" and "happiness" were only "anger" and "happiness".

[0120] Figure 8 A shows the accuracy rates of the inferred "comfort level," "arousal level," and "anger-happiness emotion." This indicates a relatively high accuracy rate of approximately 70% for each. Note that... Figure 8 A is Figure 3 The results obtained using the Fast Fourier Transform (FFT) method described in B et al. are known to yield the same results when using the Discrete Cosine Transform (DCT) (see [reference]). Figure 8 B).

[0121] As described above, emotions can also be identified using small-scale neural networks (NNs) by using pulse waves as biological information and appropriately quantizing the data. Furthermore, the NN model of one aspect of the present invention is smaller in scale and can be processed at high speed compared to the techniques disclosed in Non-Patent Documents 2 and 3, and the accuracy of emotion inference is equal to or higher than that of the techniques. That is, by using one aspect of the present invention, a small-scale, high-speed, and highly accurate emotion recognition device can be formed.

[0122] At least a portion of this embodiment can be implemented in combination with other embodiments described in this specification.

[0123] (Implementation Method 3) In this embodiment, an example of a bio-information acquisition device according to one aspect of the present invention and an electronic device using the bio-information acquisition device will be described. Note that in the bio-information acquisition device and electronic device shown in this embodiment, the same reference numerals are used to describe components having the same function.

[0124] Figure 9 A to Figure 9 Figure C illustrates an authentication device 200 that obtains pulse waves from a finger. Figure 9 Figure A is a diagram illustrating how the authentication device 200 is used. Figure 9 B is a perspective view of the authentication device 200. Figure 9 Figure C illustrates the optical sensor module 210 included in the authentication device 200.

[0125] The authentication device 200 includes a display unit 202, operation buttons 203, an input / output unit 207, and an opening 204 for inserting a finger, all housed in a housing 201. Additionally, the housing 201 includes a light sensor module 210 and a microcomputer 206.

[0126] Here, the display unit 202, the light sensor module 210, and the microcomputer 206 are respectively equivalent to Figure 1 The block diagram includes a display unit 110, a sensor 101, and a microcomputer 120. Additionally, Figure 9 The input / output section 207 shown in B is equivalent to... Figure 1 The interface is formed by integrating the input section 102 and the output section 109.

[0127] Authentication is performed by using a light sensor module 210 located on the lower side of the opening 204 to obtain photoelectric pulse waves. For example... Figure 9As shown in Figure C, the light sensor module 210 includes a light source 211 and a light-receiving element 212. The light source 211 can be, for example, an LED element that emits green light. Furthermore, the light-receiving element 212 can be a photodiode or phototransistor that is sensitive to the light emitted by the light source 211.

[0128] The light sensor module 210 is positioned at a location overlapping a portion of the finger when the finger is inserted into the opening 204. Light emitted by the light source 211 enters the interior of the finger, and the reflected light, reflected by the finger's tissue, is received by the light-receiving element 212. A portion of the light entering the finger is absorbed by the blood in the blood vessels 215; therefore, by detecting changes in the amount of reflected light, information about changes in blood vessel volume can be obtained. In other words, pulse waves that change with the heartbeat can be acquired.

[0129] Authentication device 200 can be based on Figure 2 The flowchart processes the acquired photoelectric pulse wave and outputs the personal authentication determination result to an external device through the input / output unit 207. Because the authentication device 200 of one embodiment of the present invention uses a small-scale neural network, high-speed inference can be performed using the microcomputer 206 within the authentication device 200. Therefore, authentication processing can be performed quickly and securely without connection to an AI workstation in the cloud.

[0130] Figure 9 Figure D shows an example of an authentication device 220 that obtains pulse waves from the palm. The authentication device 220 includes a display unit 202, operation buttons 203, an input / output unit 207, and a light-receiving window 213 disposed in a housing 221. In addition, a light sensor module 210 and a microcomputer 206 are included in the housing 201.

[0131] During authentication, a photoelectric pulse wave is obtained by placing the palm firmly in contact with the light-receiving window 213 and using a light sensor module 210 disposed within the frame 221 that overlaps with the light-receiving window 213. Note that the size of the light-receiving window 213 is preferably large enough to be covered by the palm to prevent ambient light from entering. Note that when a sensor capable of capturing the amount of ambient light is provided, the size of the light-receiving window 213 can also be larger than the palm. In this case, the photoelectric pulse wave can also be obtained in a non-contact manner, without close contact with the light-receiving window 213.

[0132] Authentication device 220 can be based on Figure 2The flowchart processes the acquired photoelectric pulse wave and outputs the personal authentication determination result to an external device through the input / output unit 207. Because the authentication device 220 of one embodiment of the present invention uses a small-scale neural network, high-speed inference can be performed using the microcomputer 206 within the authentication device 220. Therefore, authentication processing can be performed quickly and securely without connection to an AI workstation in the cloud.

[0133] Figure 9 Figure E is an example of a door 230 equipped with an electronic lock 232 with an authentication device. The electronic lock 232 includes an operation button 203, a door handle 233, an input / output unit 207, and a microcomputer 206. The door handle 233 is provided with a light-receiving window 213, and a photoelectric pulse wave can be obtained using a light sensor module 210 provided inside the door handle 233.

[0134] Electronic lock 232 can be based on Figure 2 The flowchart processes the acquired photoelectric pulse wave and locks or unlocks the device based on the personal authentication result. Because one embodiment of the electronic lock 232 uses a small-scale neural network, high-speed inference can be performed using the microcomputer 206 within the electronic lock 232. Therefore, authentication processing can be performed quickly and securely without connection to an AI workstation in the cloud.

[0135] Figure 9 Figure F is an example of a card 240, such as a credit card, equipped with an authentication device. Card 240 includes a microcomputer 206, an input / output unit 207, a light-receiving window 213, and a light sensor module 210. Photoelectric pulse waves can be obtained by placing a finger firmly in contact with the light-receiving window 213 on the surface of card 240 and using the light sensor module 210, which is positioned overlapping the light-receiving window 213.

[0136] Card 240 can be based on Figure 2 The flowchart processes the acquired photoelectric pulse wave and outputs the personal authentication determination result to an external device via the input / output unit 207. Alternatively, the personal authentication determination result can also be output using the wireless function of the output unit of the microcomputer 206. Because the card 240 of one embodiment of the present invention uses a small-scale neural network, high-speed inference can be performed using the microcomputer 206 within the card 240. Therefore, authentication processing can be performed quickly and securely without connection to an AI workstation in the cloud.

[0137] Figure 10Figure A shows an example of a smartphone 250 equipped with an authentication device. The smartphone 250 includes a display 202, a power button 253, operation buttons 203, a speaker 254, a microphone 256, a camera 257, a light source 258, an input / output unit 207, and a microcomputer 206, all housed in a housing 251. Alternatively, the microcomputer 206 can be replaced with components such as a CPU included in the smartphone 250.

[0138] As shown in the enlarged view of the display unit 202, in the smartphone 250, pixels may be composed of, for example, a sub-pixel R that emits red light, a sub-pixel G that emits green light, a sub-pixel B that emits blue light, and a sub-pixel S that includes a light-receiving sensor. Sub-pixels R, G, and B each include a light-emitting element.

[0139] Therefore, sub-pixel G and sub-pixel S can be used instead. Figure 9 The light sensor module 210 shown in C can acquire photoelectric pulse waves by having a part of the body, such as a finger or palm, touch the display unit 202.

[0140] Note that in Figure 10 The enlarged view of the display unit 202 shown in Figure A illustrates an example of a Delta-arranged pixel, but the pixel arrangement is not limited to this. For example, it can also be a stripe arrangement, an S-stripe arrangement, a Bayer arrangement, a zigzag arrangement, a Pentile arrangement, or a Diamond arrangement.

[0141] Alternatively, the adjacent camera 257 and light source 258 can be used instead of the light sensor module 210. In this case, photoelectric pulse waves can be obtained by placing a part of the body, such as a finger or palm, on the camera 257 and light source 258.

[0142] Here, the light emitted by the light source 258 is preferably green, but it can be white light as long as it contains a green light component. Alternatively, a structure in which the light sensor module 210 is mounted on the frame 251 can also be used.

[0143] Smartphone 250 can be based on Figure 2 The flowchart describes the processing of the acquired photoelectric pulse wave, which can utilize various functions of the smartphone 250 based on the personal authentication determination result. Because one embodiment of the smartphone 250 uses a small-scale neural network, high-speed inference can be performed using the microcomputer 206 within the smartphone 250. Therefore, authentication processing can be performed quickly and securely without connection to an AI workstation in the cloud.

[0144] Figure 10 B and Figure 10 Figure C is an example of a smartwatch 260, a watch-type electronic device equipped with an authentication device. Figure 10 B shows one side of the top surface. Figure 10 C shows the back side. The smartwatch 260 includes a display 202, operation buttons 203, input / output unit 207, and microcomputer 206, all housed in the watch case 261. Alternatively, the microcomputer 206 can be replaced with components such as a CPU included in the smartwatch 260.

[0145] The smartwatch 260 includes a light-receiving window 213 on the surface of the watch case 261 that contacts the arm, and can use a light sensor module 210 located inside the watch case 261 to obtain photoelectric pulse waves.

[0146] Smartwatch 260 can be based on Figure 2 The flowchart describes the processing of the acquired photoelectric pulse waves, which can utilize various functions of the smartwatch 260 based on the personal authentication determination result. Because one embodiment of the smartwatch 260 uses a small-scale neural network, high-speed inference can be performed using the microcomputer 206 within the smartwatch 260. Therefore, authentication processing can be performed quickly and securely without connection to an AI workstation in the cloud.

[0147] Furthermore, because wearable devices like the Smartwatch 260 are worn for extended periods, they can acquire various biological information. Therefore, in addition to authentication functions, they can also have features such as body management.

[0148] Implementation method 1 illustrates a method for identifying an individual using pulse waves, and pulse waves can also be used to infer illness or fatigue levels. Furthermore, when using a neural network (NN) model that has learned the relationship between pulse waves and brain waves, brain waves can be inferred by obtaining pulse waves, thereby confirming a person's physical state. It is generally believed that by analyzing brain waves, one can determine states such as relaxation, tension, excitement, drowsiness, and fatigue levels. Therefore, by inferring brain waves, an alert can be issued before the user becomes aware of their physical state, preventing accidents from occurring.

[0149] Note that specific equipment is required for detecting brain waves, but in one embodiment of the authentication device of this invention, such as... Figure 1 The block diagram shows that data can be input from an external device 112. As this data input, the voltage variation data obtained from the head is subjected to a Fourier transform, similar to the pulse wave, to obtain brain waves decomposed into frequency components (alpha, beta, theta, delta, and gamma waves). Alternatively, the sensor used to obtain the brain waves can also be used as... Figure 1 The block diagram shows sensor 101.

[0150] By having the neural network model learn the relationship between these brain waves and pulse waves, a person's physical condition can be indirectly inferred from the pulse waves.

[0151] Note that, although in the above Figure 9 A to Figure 9 F, Figure 10 B and Figure 10 C is shown as an authentication device, but it could also be an emotion recognition device.

[0152] For example, in Figure 9 A, Figure 9 B. Figure 10 B. Figure 10 In the structure shown in C, emotional data can be obtained by performing the same operations as the authentication device, and the stored emotional data can be used for healthcare.

[0153] In addition, Figure 10 In the smartphone 250 shown in Figure A, photoelectric pulse waves are obtained by capturing images of a part of the body, such as the face, using a camera 257, thereby enabling the recognition of the user's emotions. The recognized emotions can be used as an information source for AI-powered dialogue engines, providing information such as suggestions, images, or music tailored to the user's emotions.

[0154] Furthermore, not limited to smartphones, information processing devices including cameras, such as tablets and computers connected to cameras, can be used in stores as emotion recognition devices to analyze customers' emotions and use them for product research.

[0155] In addition, by assembling emotion recognition devices into customer service robots, responses can be tailored to customers' emotions, thereby improving customer satisfaction.

[0156] Figure 11 Figure A is an example of a customer service robot 270 equipped with an emotion recognition device. The customer service robot 270 includes a camera 271, a speaker 272, a display unit 273, a microphone 274, a driving mechanism 275, and a microcomputer 206. The display unit 273 preferably has a touchscreen function. The customer service robot uses the driving mechanism 275 to move to a position facing the customer and provides customer service through the sound emitted by the speaker 272 or the display on the display unit 273. The camera 271 is used as a sensor to acquire the customer's photoelectric pulse waves. Additionally, the display unit 273 and the microphone 274 serve as input / output units.

[0157] Customer service robot 270 can be based on Figure 2 The flowchart processes the acquired light pulse waves and responds appropriately when the customer's emotions are identified. Because the customer service robot 270 of one aspect of the present invention uses a small-scale neural network, high-speed inference can be performed using the microcomputer 206 within the customer service robot 270. Therefore, emotion recognition can be performed quickly and securely without connecting to an AI workstation in the cloud.

[0158] In addition, when emotion recognition devices are installed in vehicles such as cars that require driving, they can prevent accidents by promoting rest in a way that alleviates the driver's negative emotions or by applying external stimulation.

[0159] Figure 11 Figure B illustrates an example of an in-vehicle device 280 equipped with an emotion recognition device. The in-vehicle device 280 includes a camera 281, a speaker 282, a display unit 283, and a microcomputer 206. As an example, preferably, the camera 281, which serves as a sensor for acquiring photoelectric pulse waves, is mounted in or near the rearview mirror 285 close to the driver. Additionally, the display unit 283 may also have navigation system functionality.

[0160] The driver responds to the sounds emitted by speaker 282 or the guidelines displayed on display unit 283, and can proceed with driving operations when deemed safe. Furthermore, if the driver is determined to be experiencing negative emotions that make safe driving difficult, accidents can be prevented by promoting rest or providing external stimulation such as playing music to alleviate negative emotions or adjusting the air conditioning. Additionally, when emotions influenced by alcohol or drugs are identified, measures can be taken to prevent driving operations from beginning.

[0161] The vehicle-mounted device 280 can be based on Figure 2 The flowchart processes the acquired photoelectric pulse waves and responds appropriately based on the driver's emotional state. Because the in-vehicle device 280 of one embodiment of the present invention uses a small-scale neural network, high-speed inference can be performed using the microcomputer 206 within the in-vehicle device 280. Therefore, emotion recognition can be performed quickly and safely without connection to an AI workstation in the cloud.

[0162] At least a portion of this embodiment can be implemented in combination with other embodiments described in this specification. [Symbol Explanation]

[0163] 101: Sensor; 102: Input Unit; 103: Computation Unit; 104: Computation Unit; 105: Storage Unit; 106: Storage Unit; 107: ADC; 108: Display Driver; 109: Output Unit; 110: Display Unit; 111: AI Workstation; 112: External Device; 113: External Device; 115: Bus; 120: Microcomputer; 130: Authentication Device; 140: Emotion Recognition Device; 150: Data Quantity; 175: Data Quantity; 200: Authentication Device; 201: Frame; 202: Display Unit; 203: Operation Button; 204: Opening; 206: Microcomputer; 207: Input / Output Unit; 210: Light Sensor Module. 211: Light source; 212: Light receiving element; 213: Light receiving window; 215: Blood vessel; 220: Authentication device; 221: Frame; 230: Door; 232: Electronic lock; 233: Door handle; 240: Card; 250: Smartphone; 251: Frame; 253: Power button; 254: Speaker; 256: Microphone; 257: Camera; 258: Light source; 260: Smartwatch; 261: Watch case; 270: Customer service robot; 271: Camera; 272: Speaker; 273: Display unit; 274: Microphone; 275: Driving device; 280: In-vehicle device; 281: Camera; 282: Speaker; 283: Display unit; 285: Rearview mirror

Claims

1. A bioinformatics acquisition device, having the following functions: The function of obtaining pulse waves using simulated data; The function of converting the analog data into digital data; The function of performing a fast Fourier transform on the digital data; The function of extracting multiple data points from the low-frequency side of the data after the Fast Fourier Transform; The function to quantize each of the multiple data sets; as well as Functions for personal authentication or emotion recognition through inferences drawn from the quantized data.

2. The bioinformation acquisition device according to claim 1, The pulse wave mentioned therein is a photoelectric pulse wave.

3. The bioinformation acquisition device according to claim 2, The photoelectric pulse wave is obtained at a sampling frequency of 10 Hz or higher and 100 Hz or lower.

4. The bioinformation acquisition device according to claim 1, The number of data points extracted from the low-frequency side is between 150 and 200.

5. The bioinformation acquisition device according to claim 1, The multiple data extracted from the low-frequency side are data up to 12Hz.

6. The bioinformatics acquisition device according to claim 1, The quantized data mentioned therein is 5-value data that reflects the 4-bit representation of the flag.

7. The bioinformation acquisition device according to claim 1, The inferences made using the quantized data employ a fully connected neural network.

8. The bioinformatics acquisition device according to claim 1, The emotion recognition is performed based on either the positive or negative value of comfort or the positive or negative value of arousal.

9. The bioinformatics acquisition device according to claim 1, The emotion recognition process involves identifying the four emotions: "joy," "anger," "sorrow," and "happiness." 10. A watch-type electronic device, comprising: The bio-information acquisition device according to any one of claims 1 to 7; Case; Optical sensor module; as well as Window for receiving light The light-receiving window is located on the surface of the watch case that contacts the arm. Furthermore, the light sensor module is disposed within the watch case in a manner that overlaps with the light-receiving window.

11. The electronic device according to claim 10, The bio-information acquisition device described therein has the function of inferring a person’s physical state from the pulse wave using a neural network model that has learned the relationship between the pulse wave and brain wave.

Images