Information processing method

Abstract

To enable determination of the condition of a user.SOLUTION: An information processing method is provided, comprising causing a processor to acquire biological information of a user and determining the condition of the user according to context information when a specific waveform is detected while measuring biological information at each point in time.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing method.

Background Art

[0002] It is conceivable to operate an operation target using biological information such as brain waves.

[0003] Patent Document 1 describes a BMI device using steady-state visually evoked potential (SSVEP).

[0004] Patent Document 2 describes a BMI exercise assistance device that detects a brain wave biological signal from a user's brain waves, detects a surface electromyogram biological signal from the user's surface electromyogram, and calculates a control signal based on both biological signals.

[0005] Patent Document 3 stores in advance a brain wave condition, which is a brain wave condition that occurs when a specific movement of the body is imagined, acquires brain wave information, and when the brain waves included in the acquired brain wave information satisfy the stored brain wave condition, outputs imaging conditions, which are conditions for an imaging device to image a subject, to the imaging device. An imaging control device is described.

[0006] Patent Document 4 describes a method for matching a brain wave signal with a person's intention.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0008] An object of the present invention is to determine the state of a user.

Means for Solving the Problems

[0009] The invention according to claim 1 is an information processing method in which a processor acquires biometric information of a user and determines the state of the user according to context information when a specific waveform is measured among each time point at which the biometric information is measured.

[0010] The invention according to claim 2 is the information processing method according to claim 1, which outputs information by voice means according to the determined state of the user. The invention according to claim 3 is the information processing method according to claim 1, which executes control to call another user according to the determined state of the user. The invention according to claim 4 is the information processing method according to claim 1, which changes the setting of a function using AI according to the determined state of the user.

Effects of the Invention

[0011] According to the inventions according to claims 1-4, the state of a user can be determined.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Embodiments for Carrying Out the Invention

[0013] With reference to FIG. 1, the information processing system according to the present embodiment will be described. In FIG. 1, an example of the configuration of the information processing system according to the present embodiment is shown.

[0014] The information processing system according to the present embodiment includes an information processing apparatus 10, one or a plurality of biological information measuring apparatuses 12, one or a plurality of environmental information measuring apparatuses 14, one or a plurality of devices 16, and one or a plurality of terminal devices 18. The number of each device shown in FIG. 1 is merely an example, and the number of each device is not limited to the number of each device shown in FIG. 1. Further, the information processing system may include other devices (for example, external devices such as servers) other than the devices shown in FIG. 1. When the device 16 is not used, the device 16 may not be included in the information processing system. When the terminal device 18 is not used, the terminal device 18 may not be included in the information processing system.

[0015] Each device included in the information processing system is configured to communicate with other devices. Such communication may be wired communication using a cable or wireless communication. That is, each device may be physically connected to other devices by a cable to transmit and receive information to and from each other, or may transmit and receive information to and from each other by wireless communication. As the wireless communication, for example, short-range wireless communication, Wi-Fi (registered trademark), etc. are used. Wireless communication of other standards may also be used. The short-range wireless communication is, for example, Bluetooth (registered trademark), RFID (Radio Frequency Identifier), NFC, etc. Each device may communicate with other devices via a communication path such as a LAN (Local Area Network) or the Internet.

[0016] The information processing device 10 is, for example, a personal computer (hereinafter referred to as "PC"), a tablet PC, a smartphone, a mobile phone, or other device (for example, a server, a robot, or other equipment, etc.). The information processing device 10 may be a terminal device that can be carried by a person (for example, a tablet PC, a smartphone, a mobile phone, etc.), a device installed on a table or the like and used, a movable device (for example, a self-propelled device), a device for performing work, or a device capable of interacting with a person. For example, the information processing device 10 may be a smart speaker having a communication function, a microphone, and a speaker, or may be a robot that interacts with a person and performs work. The information processing device 10 may be a device equipped with artificial intelligence (AI).

[0017] The biological information measurement device 12 is a device configured to measure the biological information of a living organism. The living organism is a human, an animal other than a human, or a plant.

[0018] For example, the biological information measurement device 12 has various sensors, electrodes, etc., and measures the biological information of a living organism. When a plurality of biological information measurement devices 12 are used, each biological information measurement device 12 may measure different types of biological information. Some or all of the plurality of biological information measurement devices 12 may measure the same type of biological information. Also, the biological information measurement device 12 may measure one type of biological information or may measure a plurality of types of biological information.

[0019] Biological information may include various information emitted from a living organism. Biological information is, for example, information indicating brain activity (e.g., electroencephalogram, cerebral blood flow, magnetoencephalogram signal, etc.), information indicating the pulse rate, electromyographic information such as electromyogram waveforms, information regarding saliva (e.g., information indicating the amount of saliva), information indicating the pulse wave, information indicating blood pressure, information indicating blood flow, information indicating the pulse, information indicating the heart rate, information indicating the electrocardiogram waveform, information indicating eye movement, information indicating body temperature, information indicating the amount of sweating, information indicating the line of sight, voice information, information indicating the movement of a person, or information obtained from body fluids (e.g., blood, etc.). Also, information specified by a biomarker may be used as biological information. Also, biological information may be information resulting from a potential detected from a living organism. For example, biological information may be an electroencephalogram that is a measurement result of a minute current generated accompanying brain activity, an electrocardiogram that is a measurement result of a minute current generated accompanying the beating of the heart, an electromyogram that is a measurement result of a minute current generated accompanying muscle activity, or a skin potential that is a measurement result of a minute current generated on the skin. These are merely examples of biological information, and biological information other than these may be used.

[0020] By analyzing biological information, emotional information, mental information, psychological information, or the like may be obtained as human state information. For example, by analyzing a person's biological information, information indicating the person's emotion, information indicating the person's mental state, or information indicating the person's psychological state may be obtained. Of course, by analyzing the biological information of animals and plants other than humans, information indicating the states of animals and plants other than humans may be obtained. For example, as human state information, there are states of feeling the three major desires of appetite, sleep desire, and sexual desire, and consciousness mainly composed of mood. More specifically, however, it is information such as peace of mind, anxiety, gratitude, surprise, excitement, sexual excitement, curiosity, sexual curiosity, calmness, anxiety (worry), wonder (confusion), happiness, luck, relaxation, tension, honor, responsibility, respect, sense of intimacy (affection), longing (yearning), desire (will), fear, courage, pleasure, pleasure (regarding good deeds and virtues), regret, satisfaction, dissatisfaction, resentment, disgust, shame, contempt, jealousy, sense of guilt, murderous intent, schadenfreude, saudade, expectation, sense of superiority, sense of inferiority, hatred, grudge, pain, sadness, heartache, emotion, anger, trouble (anguish, remorse, distress), resignation (giving up), despair, hope, hatred (love-hate), affection, emptiness. In addition, there are also itching, pain, urinary urge, bowel movement urge, hunger, fullness, etc. Also, as communication transmission information from person to person, there is information regarding a person's thoughts and intentions. For example, approval, consent, opposition, rejection, hesitation, confusion, etc. In particular, a person's brain wave information is likely to detect a large amount of the above information compared to other biological information and is major biological information. and has a high possibility of being able to do so.

[0021] The biological information measurement device 12 may be a device (for example, a wearable device) that is attached to a living organism as a subject to measure biological information, or may be a device that measures biological information without being attached to a living organism.

[0022] Wearable devices include, for example, hearable devices worn on the ears of animals, devices worn on the heads of animals, devices worn on the wrists, arms, or fingers of animals (hereinafter referred to as "wrists, etc.") (for example, wristwatch-type devices such as smartwatches), devices worn on the necks of animals, devices worn on the bodies of animals (such as the chest and abdomen), devices worn on the lower limbs (for example, the thighs, calves, knees, feet, ankles, etc. of humans), glasses-type devices, or contact lens-type devices, etc. The biological information measuring device 12 may be a wearable device worn on other parts. Also, the biological information measuring device 12 may be worn on a plurality of parts.

[0023] A wearable device may be, for example, an earphone, a hearing aid, a pierce-type device, a clip-type device, or a device such as a band or cable that wraps around the ear. The device worn on the head may be, for example, a headset including a band or cable that wraps around the head. The device worn on the wrist or the like may be, for example, a device including a band or cable that wraps around the wrist or the like. Devices worn on other parts may also include a band, a cable, or the like. Particularly when measuring electroencephalogram information, a wearable device that can be measured with a natural appearance is preferable, and it is excellent in that it has a highly versatile form like a generally popular earphone as a device for measuring electroencephalogram. A natural appearance that does not make it apparent to others that electroencephalogram is being measured at first glance is an important factor. Medical electroencephalographs that form electrodes on the head, for example, are clearly showing that they are measuring electroencephalogram, so when worn and measured daily, for example, it becomes a major obstacle and there are many voices of reluctance to wear. In addition, since a wearable device can utilize means for inputting sound to a person's ear and outputting voice information from the mouth, it is excellent in that it can confirm the result of interpreting the information by voice while detecting electroencephalogram information, or correct the interpretation if there is an error in the interpretation. Furthermore, in BMI / BCI that mainly uses electroencephalogram, it is often used in scenes where the device is operated or communication between people is executed, but the issues are how to operate the device with high accuracy and whether the content of the communication to be conveyed can be correctly judged. Since BMI / BCI utilizing a wearable device is worn on the ear, it is possible to improve the accuracy by utilizing the above-mentioned voice input means or voice information from the mouth, or by utilizing biological information that a person can intentionally express in addition to electroencephalogram (for example, when the jaw is intentionally moved three times, it appears as electromyogram information, and when the eyeball is intentionally moved strongly, the electrooculogram information that appears, etc. can be confirmed whether it is correctly judged and used for the judgment of the implementation of the operation and the judgment of the implementation of the communication transmission).

[0024] The biological information measurement device 12 may be a contact-type device that measures the biological information of a living being by contacting the living being, or the biological information measurement device 12 may be a non-contact-type device that measures the biological information of the living being without contacting the living being. The biological information measurement device 12 may be a device that combines both a contact-type device and a non-contact-type device. That is, the biological information measurement device 12 may measure the biological information of the living being by contacting the living being, and further measure the biological information of the living being without contacting the living being. The biological information measured by contacting the living being and the biological information measured without contacting the living being may be the same type of biological information or different types of biological information.

[0025] The biological information measurement device 12 includes electrodes for measuring biological information, sensors other than electrodes, and the like. The electrode may detect a potential, which is an example of biological information, by contacting the living being, or may detect a potential without contacting the living being. Sensors other than electrodes may measure biological information by contacting the living being or may measure biological information without contacting the living being. For example, the electrode contacts an animal and detects a potential indicating the electroencephalogram of the animal. Also, the sensor may measure biological information indicating the body temperature of the animal without contacting the animal. Of course, these are only examples, and other biological information may be measured.

[0026] For example, the biological information measuring device 12 includes one or more electrodes. For example, a plurality of electrodes are provided on a living organism, and a potential is detected by the plurality of electrodes. The detected potential includes a bioelectric potential which is an example of the biological information of the living organism, and by processing the detected potential, a bioelectric potential is extracted from the detected potential. For example, the detected potential may include noise other than that derived from the bioelectric potential. By performing processing such as noise cancellation processing, a bioelectric potential from which noise has been removed can be obtained. Noise is, for example, a potential generated due to the movement of a living organism, a potential transmitted from outside the living organism, a potential generated from the earth environment, a potential representing biological information other than the measurement target, etc. A potential generated in a device such as a PC or a smartphone may be included in the noise. Depending on the detection sensitivity of the potential and the measurement situation such as noise, the electrodes used for detecting the potential may be switched. Further, when potentials derived from various bioelectric potentials are measured in a composite manner, processing may be performed to separate the potentials measured for each bioelectric potential by frequency, or during the measurement time, the composite part and the weakly composite part may be separated, so as to estimate and separate the potentials derived from various bioelectric potentials. For example, in the aforementioned wearable device, information indicating brain activity, information indicating the pulse rate, electromyographic information derived from the movement of muscles, etc., information caused by blood flow such as pulse wave and heart rate may be measured in a composite manner. However, these pieces of information often have different frequencies and absolute values of the output, and can be separated and estimated by frequency analysis of the measurement signal and the difference in the output value level. For example, in the case of electroencephalogram, it is divided into bands of α, β, θ, δ, and γ waves according to its frequency. As another method, under the same state conditions, each is measured by a wearable device, and further, a single measurement signal is individually measured a plurality of times by a commercially available electroencephalograph, electromyography measuring device, or measuring device related to blood flow. After analyzing the measurement signal information of the wearable device measured in a composite manner by a frequency analysis method such as Fourier transform or wavelet transform to determine which biological information the information is derived from, a means of introducing statistical processing to estimate and separate it may be introduced.

[0027] The biological information measurement device 12 transmits the measured biological information to the information processing device 10. Each time the biological information measurement device 12 measures biological information, it may transmit the biological information to the information processing device 10, or it may store the biological information and transmit the biological information to the information processing device 10 at predetermined time intervals, or it may transmit the biological information to the information processing device 10 at a timing specified by a person. Further, the biological information measurement device 12 may receive the biological information measured by another biological information measurement device 12 from the other biological information measurement device 12, and transmit the biological information measured by itself and the biological information measured by the other biological information measurement device 12 to the information processing device 10.

[0028] The biological information measurement device 12 may analyze the biological information measured by itself or another biological information measurement device, and transmit information indicating the result of the analysis to the information processing device 10. For example, the biological information measurement device 12 may include a processor, and the processor may analyze the biological information. Of course, the analysis may be performed by the information processing device 10 or another device.

[0029] The biological information measurement device 12 may include a battery and be driven by the power supplied from the battery, or may be driven by receiving power supply from another device (for example, the information processing device 10, etc.).

[0030] The information processing device 10 receives biological information from the biological information measurement device 12, and performs analysis of biological information, storage of biological information, output of biological information, storage of information indicating the result of analysis of biological information, and output of information indicating the result of analysis of biological information, etc. Of course, the analysis of biological information may be performed by the biological information measurement device 12 or another device. Outputting biological information is, for example, displaying biological information or outputting biological information as voice information, etc. Outputting information indicating the result of analysis of biological information is, for example, displaying information indicating the result of analysis or outputting information indicating the result of analysis as voice information, etc. The information processing device 10 may transmit biological information and information indicating the result of analysis to another device.

[0031] The information processing device 10 may include one or more biological information measurement devices 12. That is, one or more biological information measurement devices 12 may be incorporated into the information processing device 10 to form one device. The entire information processing device 10 including one or more biological information measurement devices 12 may be worn on a living organism to measure biological information. That is, the information processing device 10 may be a wearable device. For example, the information processing device 10 may be a device worn on a person's head (e.g., the forehead, etc.), a hearable device (e.g., earphones, headphones, etc.) worn on a person's ear, a device (e.g., a wristwatch-type device such as a smartwatch, etc.) worn on a person's arm, hand, wrist, finger, etc., a device hung around a person's neck, a device worn on a person's torso (e.g., abdomen, chest, etc.), or a device worn on a person's lower limbs (e.g., thigh, calf, knee, foot, ankle, etc.). The information processing device 10 may be a health device worn on a person's arm, hand, torso, or lower limbs. The information processing device 10 may be worn on other parts.

[0032] The information processing device 10 and the biological information measurement device 12 may be separate devices. For example, the information processing device 10 may be a device such as a robot, a smart speaker, or a server, and the biological information measurement device 12 may be a wearable device worn on a living organism.

[0033] The environmental information measurement device 14 is configured to measure environmental information regarding the living organism or the environment around the environmental information measurement device 14. For example, the environmental information measurement device 14 is a camera as a photographing device, a microphone for collecting sound, a temperature sensor for measuring temperature, a humidity sensor for measuring humidity, an odor sensor for measuring odor, an illuminance sensor for measuring brightness, an infrared sensor, a pressure sensor such as a barometric pressure sensor, a position sensor for detecting the movement of an object, or a human presence sensor, etc. One or more of these sensors may be included in the information processing system as the environmental information measurement device 14.

[0034] For example, the surroundings of the information processing apparatus 10 and other locations are photographed by a camera, and image data representing the surroundings and image data representing other locations are generated. The image data may be moving image data or still image data. The image data photographed by the camera corresponds to an example of environmental information regarding the environment included in the photographing range of the camera. Also, the image data generated by photographing a living organism by the camera may be used as biometric information of the living organism. For example, the movement of the living organism (e.g., a person) detected from the image data and the body shape of the living organism, etc. correspond to an example of the biometric information of the living organism. In that sense, the camera corresponds to an example of the biometric information measurement device 12 that measures the biometric information of the living organism.

[0035] Also, the sounds around the microphone (e.g., human voices and other sounds) are input into the microphone, and sound data is generated by the microphone. The sound data representing the sounds input into the microphone corresponds to an example of environmental information regarding the environment around the microphone. Also, the sound data representing the human voice input into the microphone may be used as biometric information of the person. In that sense, the microphone corresponds to an example of the biometric information measurement device 12 that measures the biometric information of the person. Also, the sound data representing the cries of animals other than humans input into the microphone may be used as biometric information of the animal.

[0036] Also, the data measured by a temperature sensor, a humidity sensor, an odor sensor, an illuminance sensor, an infrared sensor, a pressure sensor, a position sensor, a human presence sensor, etc. corresponds to an example of environmental information. Also, the data measured from a living organism by these sensors may be used as biometric information of the living organism. In that sense, these sensors correspond to an example of the biometric information measurement device 12 that measures the biometric information of the living organism.

[0037] The environmental information may be information including information indicating the size of the room where the organism is located, information indicating the size of the room where the device 16 is installed, information indicating the number of windows provided in the room, and the like. Further, the environmental information is not limited to the information measured by the environmental information measuring device 14, and may be predetermined information or information measured in advance.

[0038] Note that one or a plurality of environmental information measuring devices 14 may be included in the information processing device 10.

[0039] The device 16 is an example of an operation target. The device 16 is, for example, a PC, a tablet PC, a smartphone, a mobile phone, a robot (for example, a humanoid robot, an animal-type robot other than a human, a cleaning robot, and other robots), a projector, a display device such as a liquid crystal display, a recording device, a playback device, an imaging device such as a camera, a refrigerator, a rice cooker, a microwave oven, a coffee maker, a vacuum cleaner, a washing machine, an air conditioner, lighting equipment, a clock, a surveillance camera, an automobile, a motorcycle, an aircraft (for example, a drone), a game console, a gas range, a bidet, a ventilation fan, a doorbell, an entrance monitor, an elevator, a door, a window, or various sensing devices (for example, a temperature sensor, a humidity sensor, a voltage sensor, a current sensor, etc.). The scope of the concept of the device 16 may include all devices. For example, information devices, video devices, audio devices, and other devices may also be included in the scope of the device 16 according to the present embodiment.

[0040] The device 16 includes a communication device that is a communication interface, a memory that stores information, and a processor that controls the operation of the device 16. The device 16 may have a user interface including a display, an operation device, and the like. The device 16 may transmit device identification information for identifying the device 16 itself to the information processing device 10. The device identification information is, for example, the ID, name, model number, or address (for example, MAC address, IP address, etc.) of the device 16.

[0041] In this embodiment, an operation target is operated based on biological information of one or more organisms (for example, one or more persons). For example, the information processing apparatus 10 acquires biological information of an organism from the biological information measurement apparatus 12 and operates the operation target based on the acquired biological information. For example, the information processing apparatus 10 acquires biological information of an organism from the biological information measurement apparatus 12, specifies the operation content of the operation target based on the acquired biological information, and operates the operation target according to the specified operation content. The process of specifying the operation content may be performed by the biological information measurement apparatus 12 or another apparatus (for example, a server, the device 16, the terminal device 18, etc.). An apparatus other than the information processing apparatus 10 (for example, a server, the terminal device 18, etc.) may operate the operation target. For example, the information processing apparatus 10 may specify the operation content based on the biological information, and an apparatus other than the information processing apparatus 10 may operate the operation target according to the operation content specified by the information processing apparatus 10.

[0042] The operation target may be hardware, software, or both hardware and software.

[0043] The hardware of the operation target is not particularly limited. For example, it is the device 16 or an apparatus other than the device 16 (for example, the information processing apparatus 10, the biological information measurement apparatus 12, the environmental information measurement apparatus 14, or a server, etc.). A plurality of hardwares may be the operation target.

[0044] The software to be operated on is not particularly limited. For example, it can be an application program, an OS (Operating System), etc. The software to be operated on may be software installed in the information processing device 10, may be software installed in the device 16, may be software installed in the terminal device 18, or may be software installed in other devices (for example, the biological information measuring device 12, the environmental information measuring device 14, a server or other devices), or may be software provided via a communication path such as the Internet. A plurality of software may be the operation target. When a plurality of software are the operation target, each software may be installed in the same hardware or may be installed in different hardware.

[0045] Hereinafter, with reference to FIG. 2, the configuration of the information processing device 10 will be described in detail. FIG. 2 shows an example of the hardware configuration of the information processing device 10.

[0046] The information processing device 10 includes, for example, a communication device 20, a UI 22, a memory 24, and a processor 26. The information processing device 10 may include other configurations.

[0047] The communication device 20 is a communication interface having a communication chip, a communication circuit, etc., and has a function of transmitting information to other devices and a function of receiving information transmitted from other devices. The communication device 20 may have a wireless communication function or a wired communication function. The communication device 20 may communicate with other devices, for example, by using short-range wireless communication, or may communicate with other devices via a communication path such as a LAN or the Internet.

[0048] UI22 is a user interface and includes at least one of a display and an operating device. The display is, for example, a liquid crystal display, an EL display, or the like. The operating device is, for example, a keyboard, an input key, an operation panel, or the like. UI20 may be a UI such as a touch panel that combines a display and an operating device. UI20 may include a microphone and a speaker. Note that the information processing apparatus 10 may not include UI20.

[0049] The memory 24 is a device that constitutes one or more storage areas for storing various types of information. The memory 24 is, for example, a hard disk drive, various memories (e.g., RAM, DRAM, ROM, etc.), other storage devices (e.g., optical disks, etc.), or a combination thereof. One or more memories 24 are included in the information processing apparatus 10.

[0050] Management information is stored in the memory 24. The management information is information for specifying the operation content of the operation target based on biological information.

[0051] The processor 26 is configured to control the operations of each part of the information processing apparatus 10. The processor 26 may have a memory.

[0052] For example, predetermined reference biological information and operation information indicating the operation content of the operation target are associated in advance and registered in the above management information. The reference biological information may be biological information presumed to be generated from an animal (e.g., a person or other animal) that performs an operation associated with the reference biological information, or biological information presumed to be generated from an animal that requests or desires the execution of the operation, or biological information presumed to be generated from a plant. It can also be said that the reference biological information is biological information indicating the operation content of the operation target. The operation information indicating the operation content may include information indicating the operation target to be operated according to the operation content.

[0053] For each living organism, reference biological information and operation information may be associated and registered in management information. For example, for each type of living organism, reference biological information and operation information may be associated and registered in management information. For example, management information for humans, management information for animals other than humans, and management information for plants may be created, and each management information may be stored in the memory 24. In the management information for humans, reference biological information and operation information that are presumed to be measured from humans are associated and registered. For each individual human (for example, for each individual user), reference biological information and operation information may be associated and registered in management information. In the management information for animals other than humans, reference biological information and operation information that are presumed to be measured from the animal are associated and registered. Management information may be created for each type of animal other than humans. Also, in the management information for plants, reference biological information and operation information that are presumed to be measured from plants are associated and registered. Management information may be created for each type of plant.

[0054] The operation information may be information indicating the operation content of the power supply of the device 16, or may be information indicating the operation content of the function level of the device 16, or may be information indicating both the operation content of the power supply and the operation content of the function level.

[0055] The operation content of the power supply of the device 16 is an operation to turn on or off the power supply of the device 16. The operation information regarding the operation content of the power supply is information indicating the operation to turn on or off the power supply of the device 16. It can be said that the biological information associated with the operation information regarding the operation content of the power supply is the biological information corresponding to the turning on or off of the power supply of the device 16. For each type of living organism or for each individual living organism, operation information regarding the operation content of the power supply of the device 16 and reference biological information may be associated and registered in management information.

[0056] The operation content at the function level of device 16 is an operation for setting the function level of device 16. The operation information regarding the operation content at the function level is information indicating the operation for setting the function level of device 16. It can be said that the biological information associated with the operation information regarding the operation content at the function level is the biological information corresponding to the function level of device 16. For each type of organism or each individual organism, the operation information regarding the operation content at the function level of device 16 and the reference biological information may be associated and registered in the management information.

[0057] The function level is, for example, a level regarding the performance or output of device 16. To explain with specific examples, the set temperature of an air conditioner, the air volume of an air conditioner, the air direction of an air conditioner, the presence or absence of the dehumidifying function of an air conditioner, the brightness of a display device, the brightness of a lighting device, the volume of a speaker, the moving speed of a self - propelled device (such as a robot or a self - propelled vacuum cleaner, etc.), the set values of devices such as a photographing device, a recording device, a playback device, etc., the set values of devices such as a refrigerator, a rice cooker, a microwave oven, etc., and the set values of various sensing devices, etc. correspond to an example of the device level. These are just examples, and other set values, etc. may also be the function level.

[0058] Note that the management information may not be stored in the information processing device 10, or may be stored in the information processing device 10 and also stored in a device other than the information processing device 10.

[0059] The processor 26 is configured to acquire the biological information of an organism and operate an operation target based on the biological information.

[0060] For example, when biological information is measured from an organism by the biological information measurement device 12, the biological information is transmitted from the biological information measurement device 12 to the information processing device 10. The processor 26 receives the biological information and identifies the operation content of the operation target based on the biological information. The processor 26 operates the operation target according to the identified operation content. The processor 26 may identify the operation target and the operation content of the operation target based on the biological information of the organism. As another example, the operation target may be predetermined. For example, the operation target may be specified by a person. In this case, the processor 26 identifies the operation content of the defined operation target based on the biological information of the organism.

[0061] When the operation target is the device 16, the processor 26 operates the device 16 by transmitting control information including operation information indicating the identified operation content to the device 16. The device 16 as the operation target operates according to the control information. The same applies when the hardware other than the device 16 is the operation target.

[0062] When the operation target is software, the processor 26 operates the software according to the identified operation content. For example, the processor 26 starts the software or performs processing using the software. When the software is installed in the information processing device 10, the processor 26 operates the software installed in the information processing device 10. When the software is installed in a device other than the information processing device 10, the processor 26 operates the software by transmitting control information including operation information indicating the operation content to the device in which the software is installed.

[0063] For example, the processor 26 compares the biological information measured from a living organism with each reference biological information registered in the above management information, and searches for the reference biological information whose difference from the biological information is within the allowable range. The processor 26 specifies the operation content for the operation target associated with the searched reference biological information. Thereby, the operation target and the operation content of the operation target are specified. The allowable range is predetermined. The allowable range may be changed by the user. The allowable range may be determined for each type of living organism or for each individual living organism.

[0064] When a plurality of reference biological information whose differences from the biological information measured from a living organism are within the allowable range are searched, the processor 26 may specify the reference biological information with the minimum difference from the biological information among the plurality of reference biological information, and specify the operation content associated with the specified reference biological information, or may specify the operation content associated with each of the plurality of reference biological information.

[0065] The reference biological information may be information indicating a characteristic component of the biological information. In this case, the processor 26 may extract the characteristic component from the biological information measured from the living organism, and search for the reference biological information having the component whose difference from the extracted component is within the allowable range. For example, when electroencephalogram is used as the biological information, the processor 26 may extract the characteristic component from the electroencephalogram of the animal, and analyze the component to infer the operation content represented by the electroencephalogram.

[0066] Reference biological information may be used as a threshold value. For example, when the reference biological information shows a waveform, the amplitude of the peak in the waveform is used as the threshold value. In this case, the processor 26 searches for reference biological information having a waveform similar or analogous to the waveform shown by the biological information measured from the organism, and the reference biological information for which the measured biological information is equal to or greater than the threshold value, and identifies the operation content associated with the searched reference biological information. That is, when the waveform shown by the measured biological information is similar or analogous to the waveform shown by a certain reference biological information, and the amplitude of the peak of the waveform shown by the measured biological information is equal to or greater than the threshold value which is the amplitude of the peak of the waveform shown by the reference biological information, the processor 26 identifies the operation content associated with the reference biological information.

[0067] When operating an operation target using information indicating the activity of an animal's brain, a brain-machine interface (BMI) may be constructed by a biological information measurement device 12 that measures brain activity and an information processing device 10. Generally, a brain-machine interface is a general term for devices that interface between the brain and a computer or the like by means of detecting brain waves or the like, or conversely, stimulating the brain. When the connection destination is a computer, it is also called a brain-computer interface (BCI). In the present embodiment, examples of utilization such as operating a device or communicating between people (mutual transmission of information and state information regarding a person's thoughts and intentions) by utilizing biological information including brain waves are shown, but these are regarded as part of the brain-machine interface and will be hereinafter referred to as such. The method of the brain-machine interface may be invasive or non-invasive. In this case, the processor 26 operates the operation target based on the activity of the brain (for example, brain waves). The processor 26 may extract characteristic components from the brain waves and operate the operation target based on the extracted components in order to operate the operation target. In order to extract characteristic components from the brain waves, fast Fourier transform (FFT), wavelet transform (WT), time-frequency distribution (TFD), eigenvector methods (EM), autoregressive model (ARM), or the like may be used. Further, as a method of associating the brain waves with the operation content of the operation target or as a method of associating the transmission target with the transmission content in the communication transmission between people using the feature vector obtained by the extraction of features, for example, independent component analysis (ICA), k-means method, support vector machine (SVM), convolutional neural network, or the like may be used.

[0068] Further, the processor 26 may receive the identification information of the operation target transmitted from the operation target and identify the operation target. For example, the processor 26 transmits a request for acquiring identification information to the operation target and acquires the identification information transmitted from the operation target in response to the request. Further, the identification information of the hardware that has become capable of communicating with the information processing apparatus 10 by being connected to the information processing apparatus 10 or the like, and the identification information of the software installed in the hardware may be transmitted to the information processing apparatus 10. The processor 26 receives the identification information transmitted in such a manner.

[0069] Note that the processing by the processor 26 may be executed by a device other than the information processing apparatus 10 (for example, the biological information measurement device 12, the environmental information measurement device 14, the device 16, the terminal device 18, or other devices). Further, a part of the processing may be executed by the information processing apparatus 10, and another part of the processing may be executed by a device other than the information processing apparatus 10. For example, the analysis of biological information (for example, the process of specifying the operation content) may be executed by a device other than the information processing apparatus 10, and the operation of the operation target may be executed by the information processing apparatus 10.

[0070] Hereinafter, with reference to FIG. 3, the configuration of the terminal device 18 will be described in detail. FIG. 3 shows an example of the hardware configuration of the terminal device 18.

[0071] The terminal device 18 includes, for example, a communication device 28, a UI 30, a memory 32, and a processor 34.

[0072] The communication device 28 is a communication interface having a communication chip, a communication circuit, etc., and has a function of transmitting information to other devices and a function of receiving information transmitted from other devices. The communication device 28 may have a wireless communication function or a wired communication function. The communication device 28 may communicate with other devices, for example, by using short-range wireless communication, or may communicate with other devices via a communication path such as a LAN or the Internet.

[0073] UI30 is a user interface and includes at least one of a display and an operating device. The display is, for example, a liquid crystal display, an EL display, or the like. The operating device is, for example, a keyboard, an input key, an operation panel, or the like. UI30 may be a UI such as a touch panel that combines a display and an operating device. UI30 may also include a microphone and a speaker.

[0074] Memory 32 is a device that constitutes one or more storage areas for storing various types of information. Memory 32 is, for example, a hard disk drive, various types of memories (such as RAM, DRAM, ROM, etc.), other storage devices (such as optical disks, etc.), or a combination thereof. One or more memories 32 are included in the terminal device 18.

[0075] Processor 34 is configured to control the operations of each part of the terminal device 18. Processor 34 may have a memory.

[0076] Hereinafter, specific examples of management information will be described.

[0077] FIG. 4 shows an example of a management table, which is an example of management information. The data of the management table is stored in the memory 24. The data of the management table may be stored not in the memory 24 but in another device (such as a server device) other than the information processing device 10.

[0078] In the management table, an ID, a reference brain wave, and operation information indicating the operation content of the operation target are associated in advance.

[0079] The ID is information for managing the information registered in the management table.

[0080] The reference brain wave is an example of reference biological information. Also, the reference brain wave is a brain wave that is presumed to be measured from a person. Here, brain waves are used as an example of reference biological information, but biological information other than brain waves may be used as reference biological information.

[0081] The reference brain wave is determined, for example, by statistical processing, and is a brain wave that is presumed to be generated from a person who performs the operation content associated with the reference brain wave, or a brain wave that is presumed to be generated from a person who requests the execution of the operation content.

[0082] The reference brain wave may be a brain wave in a specific frequency band, or may be a brain wave including brain waves in a plurality of frequency bands.

[0083] In the example shown in FIG. 4, the operation target is the device 16 which is an example of hardware. Of course, the operation target may be hardware other than the device 16, or may be software.

[0084] The operation information is information including device identification information for identifying the device 16 to be operated and information indicating the operation content performed on the device 16. For example, the operation content may be an operation to turn on or off the power of the device 16, an operation to set the function level of the device 16, or both an operation to turn on or off the power and an operation to set the function level.

[0085] For example, the reference brain wave of ID "1" is a brain wave representing the operation content of turning on the cooling of the air conditioner. The reference brain wave of ID "2" is a brain wave representing the operation content of turning off the cooling of the air conditioner.

[0086] For example, when a brain wave whose difference from the reference brain wave of ID "1" is within the allowable range is measured from a person, the processor 26 identifies the operation content of "turning on the cooling of the air conditioner" associated with the reference brain wave. Then, the processor 26 transmits control information including information indicating the operation content of "turning on the cooling of the air conditioner" to the air conditioner. The air conditioner operates according to the control information. As a result, the cooling of the air conditioner is turned on.

[0087] In addition, the processor 26 may calculate the similarity between the brain waves measured from a person and the reference brain waves, and determine whether the similarity is equal to or greater than a threshold value. The threshold value is a value corresponding to the allowable range. When the similarity between the brain waves measured from a person and the reference brain waves is equal to or greater than the threshold value, the processor 26 determines that the brain waves measured from the person are similar to the reference brain waves. That is, the processor 26 determines that the difference between the brain waves measured from the person and the reference brain waves is within the allowable range. When brain waves with a similarity equal to or greater than the threshold value with the reference brain waves of ID "1" are measured from a person, the processor 26 identifies the operation content of "turn on the cooling of the air conditioner".

[0088] Note that in the example shown in FIG. 4, the reference brain waves, which are an example of biological information, are registered in the management table. However, emotional information, mental information, psychological information, etc. obtained by analyzing biological information may be registered in the management table instead of the reference brain waves. For example, information indicating emotions, mental states, or psychological states such as "hot" or "cold" may be registered in the management table as reference information instead of the reference brain waves. The processor 26 analyzes the brain waves measured from a person to identify the emotions, mental states, or psychological states of the person, identifies the reference information indicating the identified emotions, mental states, or psychological states, and in the management table, identifies the operation content associated with the identified reference information.

[0089] To explain with a specific example, reference information indicating an emotion, mental state, or psychological state of "hot" is registered in the management table in association with operation information indicating the operation content of "turn on the cooling of the air conditioner". The processor 26 analyzes the biological information (e.g., brain waves) measured from a person to identify the emotions, mental states, or psychological states of the person. When the emotion or the like is an emotion or the like of "hot", the processor 26 identifies the operation content of "turn on the cooling of the air conditioner" associated with the reference information of "hot".

[0090] In this way, the processor 26 may identify the operation content based on the biometric information itself, or may identify the operation content based on various types of information obtained from the biometric information (for example, emotion information, mental information, psychological information, etc.). The same applies to each of the following embodiments.

[0091] For each living organism, reference biometric information and operation information may be associated and registered in a management table. For example, biometric information measured from an individual person may be registered in the management table as the reference biometric information of that person. For animals and organisms other than humans, biometric information measured from individual animals or individual organisms may also be registered in the management table as the reference biometric information of that animal or that organism.

[0092] FIG. 5 shows an example of a management table in which specific reference biometric information of individual persons is registered. In the management table shown in FIG. 5, an ID, a reference electroencephalogram which is an example of reference biometric information, operation information, and user information are associated. The user information is information for identifying a user who is a person (for example, a user name, a user ID, etc.).

[0093] The reference electroencephalogram associated with the user information is the electroencephalogram measured from the user when the user indicated by the user information performs the operation content associated with the reference electroencephalogram, or the electroencephalogram measured from the user when the user requests the operation content. Each reference electroencephalogram measured from each user is measured from each user in advance and registered in the management table.

[0094] For example, when user A manually turns on the cooling of the "air conditioner", the brain waves of user A are measured by the biological information measuring device 12, and the measured brain waves are registered in the management table as the reference brain waves representing the operation content of "turning on the cooling of the air conditioner" by user A. In this case, the measured reference brain waves of user A, the operation information indicating the operation content of "turning on the cooling of the air conditioner", and the user information for identifying user A are associated and registered in the management table. The registration may be performed using the information processing device 10 or using other devices. In the example shown in FIG. 5, these pieces of information are registered as the information with ID "1". The same applies to other operations and other users.

[0095] Note that the registration work may be performed multiple times, and the average of the multiple brain waves measured thereby may be registered as the reference brain wave. For example, the work of user A manually turning on the cooling of the "air conditioner" and measuring the brain waves generated by user A at that time by the biological information measuring device 12 may be performed multiple times, and the average of the multiple brain waves measured thereby may be registered in the management table as the reference brain wave of user A.

[0096] For example, when a brain wave whose difference from the reference brain wave of ID "1" is within the allowable range is measured from user A while user A is logged in to the information processing apparatus 10, the processor 26 transmits control information including the operation information of ID "1" to the "air conditioner", thereby turning on the cooling of the "air conditioner". More specifically, when a brain wave is measured by the biological information measuring apparatus 12 while user A is logged in to the information processing apparatus 10, the processor 26 searches for the reference brain wave registered in the management table associated with the user information for identifying the logged-in user A. In the example shown in FIG. 5, since the reference brain wave of ID "1" and the reference brain wave of ID "3" are registered in the management table as the reference brain wave of user A, these reference brain waves are searched. When the difference between the measured brain wave and the reference brain wave of ID "1" is within the allowable range, the processor 26 transmits control information including the operation information of ID "1" to the "air conditioner", thereby turning on the cooling of the "air conditioner". When the difference between the measured brain wave and the reference brain wave of ID "3" is within the allowable range, the processor 26 transmits control information including the operation information of ID "3" to the "air conditioner", thereby turning off the cooling of the "air conditioner".

[0097] As another example, when a brain wave whose difference from the reference brain wave of ID "1" is within the allowable range is measured from user A while it is set in the information processing apparatus 10 that the user operating the device 16 is "user A", the processor 26 may transmit control information including the operation information of ID "1" to the "air conditioner", thereby turning on the cooling of the "air conditioner". More specifically, when a brain wave is measured by the biological information measuring apparatus 12 while it is set in the information processing apparatus 10 that the user operating the device 16 is "user A", the processor 26 searches for the reference brain wave registered in the management table associated with the user information for identifying user A who is the user operating the device 16. When the difference between the measured brain wave and the reference brain wave of ID "1" is within the allowable range, the processor 26 transmits control information including the operation information of ID "1" to the "air conditioner", thereby turning on the cooling of the "air conditioner". The user who operates the device 16 is set in the information processing apparatus 10 by the user, for example.

[0098] For other users other than User A, each piece of information is registered in the management table in the same manner as User A. For example, each piece of information associated with ID "2" is information regarding the operation when User B turns on the cooling of the "air conditioner". Each piece of information associated with ID "3" is information regarding the operation when User A turns off the cooling of the "air conditioner".

[0099] In the management table shown in FIGS. 4 and 5, operation information indicating an operation to turn on or off the power of device 16 is registered, but operation information indicating the function level of device 16 may also be registered in the management table.

[0100] FIG. 6 shows another example of the management table. The management table shown in FIG. 6 is a management table in which reference biological information and operation information are associated and registered for each user. In the example shown in FIG. 6, as an example, brain waves and body temperature are used as reference biological information. Therefore, in the management table, an ID, reference brain waves which are an example of reference biological information, reference body temperature which is an example of reference biological information, operation information, and user information are associated with each other.

[0101] For example, when brain waves whose difference from the reference brain waves of ID "1" is within the allowable range are measured from User A and a body temperature equal to or higher than the threshold value is measured from User A, the processor 26 specifies the operation content indicated by the operation information of ID "1" as the operation content for device 16. Then, the processor 26 transmits control information including the operation information indicating the operation content of ID "1" to the air conditioner to turn on the cooling of the air conditioner.

[0102] Note that in the example shown in FIG. 6, reference body temperature, reference brain waves, and operation information are associated for each person, but reference body temperature, reference brain waves, and operation information may be associated not for each person.

[0103] FIG. 7 shows another example of the management table. The management table shown in FIG. 7 is a management table in which a reference electroencephalogram, reference environmental information, and operation information are registered in association with each user. Although a reference electroencephalogram is used as an example of the reference biological information, another biological information may be used as the reference biological information.

[0104] The reference environmental information is information indicating an environment in which it is presumed that the reference electroencephalogram associated with the reference environmental information is measured. Although room temperature is used as the reference environmental information, another environmental information may be used as the reference environmental information.

[0105] For example, when an electroencephalogram whose difference from the reference electroencephalogram of ID "1" is within the allowable range is measured from user A, and the room temperature when the electroencephalogram is measured is equal to or higher than the threshold value, the processor 26 identifies the operation content indicated by the operation information of ID "1" as the operation content for the device 16. In this way, based on the electroencephalogram measured from the user and the room temperature when the electroencephalogram is measured, the operation content for the device 16 is identified.

[0106] In the example shown in FIG. 7, the reference biological information, the reference environmental information, and the operation information are associated with each person, but the reference biological information, the reference environmental information, and the operation information may be associated not with each person.

[0107] Hereinafter, each embodiment will be described.

[0108] <Example 1> Hereinafter, Example 1 will be described. In Example 1, biological information measured from a plurality of organisms in a certain place in the real space is used. Here, as an example, an example of operating the device 16 based on the biological information measured from a plurality of people gathered in a certain room will be described.

[0109] FIG. 8 schematically shows the room 36. In the room 36, there are users A, B, and C who are people. Also, the device 16 is installed in the room 36. Note that the number of users and the device 16 is an example, and a plurality of devices 16 may be installed in the room 36.

[0110] The biological information of each user is measured by the biological information measuring device 12. For example, the biological information measuring device 12, which is a wearable device, is worn by each of users A, B, and C, and the respective biological information of users A, B, and C is measured by the biological information measuring device 12 worn by each user. The biological information measuring device 12 may be worn by each user, or the biological information of each user may be measured by the biological information measuring device 12 that is not worn by the user without being worn by each user. One or more types of biological information may be measured from each user. Here, as an example, it is assumed that electroencephalograms are measured from each of users A, B, and C. Of course, biological information other than electroencephalograms may be measured.

[0111] For example, the biological information (for example, electroencephalogram signal) of the users in the room 36 is transmitted from the biological information measuring device 12 of each user to the information processing device 10. In the example shown in FIG. 8, the respective biological information (for example, electroencephalogram signal) of users A, B, and C is transmitted from the biological information measuring device 12 of each user to the information processing device 10.

[0112] The processor 26 of the information processing device 10 operates the device 16 based on the electroencephalograms of each of users A, B, and C. For example, the processor 26 calculates the average of the electroencephalograms measured from each of users A, B, and C, and specifies the operation content of the device 16 based on the average electroencephalogram. For example, the processor 26 refers to the management table shown in FIG. 4, specifies the reference electroencephalogram whose difference from the average electroencephalogram is within the allowable range, and specifies the operation content associated with the specified reference electroencephalogram. Then, the processor 26 operates the device 16 according to the specified operation content. For example, the processor 26 calculates the average of the amplitudes of the electroencephalogram, specifies the reference electroencephalogram whose difference from the electroencephalogram having the average amplitude is within the allowable range, and specifies the operation content.

[0113] As another example, when the operation content of device 16 is the setting of the function level, the processor 26 may separately identify the function levels of device 16 based on the brain waves of users A, B, and C respectively. Then, the processor 26 calculates the average of each function level and operates device 16 according to the average function level. For example, when device 16 is an air conditioner having a cooling function, the operation content identified based on the brain wave of user A is the operation content of "setting the set temperature of the cooling of the air conditioner to 26 °C", and the operation content identified based on the brain wave of user B is the operation content of "setting the set temperature of the cooling of the air conditioner to 27 °C", and the operation content identified based on the brain wave of user C is the operation content of "setting the set temperature of the cooling of the air conditioner to 28 °C", the processor 26 calculates the average of these set temperatures. Then, the processor 26 sets the set temperature of the cooling of the air conditioner to the average temperature.

[0114] When the operation content of device 16 is an operation to turn device 16 on or off, the processor 26 may separately identify the on or off of device 16 based on the brain waves of users A, B, and C respectively. Then, the processor 26 determines the on or off of device 16 according to the majority vote of users A, B, and C, and turns device 16 on or off according to the determination.

[0115] (Operation based on the distance between device 16 and the user) The processor 26 of the information processing apparatus 10 may identify the operation content of device 16 based on the biological information (for example, brain wave signal) of the user whose distance from the device 16 to be operated is equal to or less than the threshold value, and operate device 16 according to the operation content. The positions of device 16 and each user are identified, for example, by using GPS (Global Positioning System) or other communication technologies, etc., and the position information of device 16 and each user is transmitted to the information processing apparatus 10. For example, when the user carries the biological information measuring device 12 or the terminal device 18, the positions of the biological information measuring device 12 and the terminal device 18 are identified by using GPS or the like, and the position information indicating the positions is transmitted to the information processing apparatus 10 as the position information indicating the position of the user.

[0116] For example, when the distances between users A and B and device 16 are equal to or less than a threshold value and the distance between user C and device 16 exceeds the threshold value, the processor 26 of the information processing apparatus 10 calculates the average of the brainwaves of users A and B, specifies the operation content of device 16 based on the average brainwaves, and operates device 16 according to the specified operation content. Even when the brainwave signal of user C is transmitted to the information processing apparatus 10, the processor 26 specifies the operation content of device 16 without using the brainwaves of user C. By doing so, device 16 can be operated based on the biometric information of the users whose distances from device 16 are equal to or less than the threshold value.

[0117] (Operation based on the distance between the information processing apparatus 10 and the user) The processor 26 of the information processing apparatus 10 may specify the operation content of device 16 based on the biometric information of the users whose distances from the information processing apparatus 10 are equal to or less than the threshold value, and operate device 16 according to the operation content. For example, when the information processing apparatus 10 is a device such as a smart speaker and is installed in room 36, the processor 26 may operate device 16 based on the biometric information of the users whose distances from the information processing apparatus 10 are equal to or less than the threshold value.

[0118] (Weighting process) The processor 26 of the information processing apparatus 10 may perform a weighting process on the brainwaves of each user, calculate the average of the weighted brainwaves, and specify the operation content of device 16 based on the average brainwaves.

[0119] For example, a weighting coefficient is determined for each user, and information indicating the weighting coefficient of each user is stored in the memory 24 of the information processing apparatus 10. For example, each user may determine their own weighting coefficient, or a user selected from users A, B, and C may determine the weighting coefficient of each user. Information indicating the weighting coefficient of each user may be transmitted from the terminal device 18 of each user to the information processing apparatus 10, or may be input into the information processing apparatus 10 using the UI 22 of the information processing apparatus 10.

[0120] The weight coefficient of each user may be determined based on the attributes of the user. The attributes are, for example, age, gender, height, weight, etc. The weight coefficient of each attribute is predetermined, and information indicating the weight coefficient of each attribute is stored in advance in the memory 24 of the information processing apparatus 10. For example, information indicating the attributes of each user is transmitted from each user's terminal device 18 to the information processing apparatus 10, and the processor 26 of the information processing apparatus 10 determines the weight coefficient of each user based on the information indicating the attributes of each user.

[0121] Also, the weight coefficient may be determined based on the distance between the device 16 and the user. For example, the closer the distance between the device 16 and the user, the larger the weight coefficient, and the longer the distance, the smaller the weight coefficient. Of course, the reverse may also be true. The distance is calculated based on the position information of the device 16 and the user respectively.

[0122] (When the biological information changes) When the brain waves of users A, B, and C change respectively, the processor 26 of the information processing apparatus 10 changes the operation content of the device 16 according to the change. For example, the processor 26 calculates the average of the brain waves of users A, B, and C every predetermined period, specifies the operation content for each period based on the average, and operates the device 16 for each period according to the specified operation content. Also in this case, the above-described weighting process may be performed.

[0123] In addition, in the first embodiment, the plurality of organisms for which biological information is measured may be at least two combinations of a human, an animal other than a human, and a plant. For example, the device 16 may be operated based on the biological information of each of a human and an organism. The same applies to the embodiments described below.

[0124] (Processing when the organism moves) Hereinafter, with reference to FIG. 9, the processing when the organism moves will be described. In FIG. 9, a room 36 is schematically shown. Similar to the example shown in FIG. 8, in the room 36, there are users A, B, and C who are humans. Also, a device 16 is installed in the room 36.

[0125] For example, as indicated by the arrow in FIG. 9, when user A moves and the distance between user A and device 16 exceeds a threshold value while the distances between users B and C and device 16 are below the threshold value, the processor 26 of the information processing apparatus 10 identifies the operation content based on the brainwaves of users B and C and operates device 16 according to the identified operation content. As described above, the processor 26 may identify the operation content based on the average of the brainwaves of users B and C and operate device 16 according to the identified operation content, or may identify the function level based on the respective brainwaves of users B and C and operate device 16 according to the average of the identified function levels, or may determine on or off based on the respective brainwaves of users B and C and turn device 16 on or off according to the determination.

[0126] The processor 26 may perform the above-described weighting process. In this case, if the user A before moving is a user who affects the determination of the operation content (for example, when the weighting coefficient of user A is larger than the weighting coefficients of other users), the operation content of device 16 may change when user A moves.

[0127] Note that even when biometric information of animals or organisms other than humans is measured, if the non-human animal moves or the arrangement position of the organism changes, the operation content of device 16 may be changed accordingly.

[0128] <Example 2> Hereinafter, Example 2 will be described. In Example 2, biometric information measured from a plurality of organisms located in different places is used. Here, as an example, an example of operating device 16 based on biometric information measured from people located in different places (for example, places that are not in the same room) will be described. The scenario to which Example 2 is applied is, for example, a scenario where a plurality of people conduct conversations, meetings, etc. online (for example, remote meetings, web conferences, etc.).

[0129] For example, assume that users A, B, C, and D, who are humans, are participating in an online meeting. User A uses terminal device 18A, user B uses terminal device 18B, user C uses terminal device 18C, and user D uses terminal device 18D. Terminal devices 18A, 18B, 18C, and 18D have the same configuration as terminal device 18.

[0130] Terminal devices 18A, 18B, 18C, and 18D can receive the provision of such a service from a server that provides a service for online conversation or meeting, and can exchange information (such as images, voices, character strings, etc.) with each other via the server. As such a service, for example, a known service is used.

[0131] Hereinafter, with reference to FIG. 10, Example 2 will be described in detail. FIG. 10 shows a screen 38A. Screen 38A is a screen displayed on the display of UI 30 of terminal device 18A of user A.

[0132] Images of users A, B, C, and D participating in the online meeting are displayed on screen 38A. For example, screen 38A includes regions 40A, 40B, 40C, and 40D. An image of user A is displayed in region 40A, an image of user B is displayed in region 40B, an image of user C is displayed in region 40C, and an image of user D is displayed in region 40D. Screens having the same configuration as screen 38A are also displayed on the respective displays of terminal devices 18B, 18C, and 18D.

[0133] For example, user A is photographed by a camera provided in terminal device 18A of user A or a camera not provided in terminal device 18A, and the processor 34 of terminal device 18A displays the image generated by the photographing in region 40A. Also, the image data of user A is transmitted from terminal device 18A to terminal devices 18B, 18C, and 18D via the server, and an image of user A is displayed on the respective displays of terminal devices 18B, 18C, and 18D. The same applies to the respective images of users B, C, and D.

[0134] Similar to Example 1, the biological information of each user is measured by the biological information measuring device 12. Here, as an example, similar to Example 1, it is assumed that electroencephalograms are measured from each of users A, B, C, and D. Of course, biological information other than electroencephalograms may be measured. The biological information (for example, electroencephalogram signals) of each of users A, B, C, and D is transmitted from the biological information measuring device 12 of each user to the information processing device 10.

[0135] The processor 26 of the information processing device 10 operates the device 16 based on the electroencephalograms of each of users A, B, C, and D. The device 16 to be operated may be the terminal devices 18A, 18B, 18C, and 18D, or may be a device other than the terminal device 18.

[0136] Each terminal device 18 may have the functions of the information processing device 10, and the processor 34 of each terminal device 18 may operate the device 16 to be operated. For example, each terminal device 18 receives biological information from the biological information measuring device 12 of each user via a communication path and operates the device 16.

[0137] For example, the processor 34 of each terminal device 18 controls the volume of the speaker provided in the terminal device 18 based on the electroencephalogram of each user. For example, when it is determined that user B is excited by analyzing the electroencephalogram of user B, the processor 34 of each terminal device 18 controls the volume of the speaker of each terminal device 18 so that the volume of user B becomes smaller. For example, the processor 34 of the terminal device 18A controls the volume of the speaker of the terminal device 18A so that the volume of user B becomes smaller. The same applies to the other terminal devices 18. When a speaker not provided in the terminal device 18 is used, the processor 34 controls the volume of the speaker.

[0138] Note that authentication of users participating in the online meeting may be performed. Users who have successfully authenticated can participate in the online meeting. For example, authentication technologies such as face authentication, voice authentication, retina authentication, or otoacoustic emission authentication may be used. The authentication may be performed by the information processing apparatus 10 or by an authentication server.

[0139] Also in the second embodiment, the processor 26 of the information processing apparatus 10 may operate the device 16 based on the brain waves of each user (for example, based on the average of the brain waves). The device 16 to be operated may be a device used by at least one of users A, B, C, and D, or may be another device. In this case, weighting processing may be performed.

[0140] (Processing when the living being moves) Hereinafter, with reference to FIG. 11, the processing when the living being moves will be described. FIG. 11 shows a screen 38A.

[0141] For example, when user B who was participating in the online meeting stops participating in the online meeting or when the image of user B is no longer displayed in area 40B, the processor 26 of the information processing apparatus 10 operates the device 16 based on the brain waves of each of users A, C, and D.

[0142] <Example 3> Hereinafter, with reference to FIG. 12, Example 3 will be described. Example 3 is a combination of Examples 1 and 2. FIG. 12 shows a screen 38A.

[0143] For example, users A1 and A2 are participating in an online meeting using terminal device 18A, users B1 and B2 are participating in an online meeting using terminal device 18B, user C1 is participating in an online meeting using terminal device 18C, and users D1, D2, and D3 are participating in an online meeting using terminal device 18D. For example, users A1 and A2 belong to group A, users B1 and B2 belong to group B, user C1 belongs to group C, and users D1, D2, and D3 belong to group D. Information regarding the groups (for example, group identification information and identification information of each user belonging to the group, etc.) is stored in information processing device 10, a server, or the like.

[0144] Images of users A1 and A2 are displayed in area 40A, images of users B1 and B2 are displayed in area 40B, an image of user C1 is displayed in area 40C, and images of users D1, D2, and D3 are displayed in area 40D.

[0145] Similar to Example 1, the biometric information of each user is measured by biometric information measurement device 12. Note that the biometric information of one or more users participating in the online meeting may not be measured. Here, as an example, similar to Example 1, electroencephalograms are measured from each user. Of course, biometric information other than electroencephalograms may be measured. The biometric information (for example, electroencephalogram signals) of each user is transmitted from the biometric information measurement device 12 of each user to information processing device 10.

[0146] Similar to Example 2, the processor 26 of information processing device 10 operates device 16 based on the electroencephalograms of each user. The device 16 to be operated may be terminal devices 18A, 18B, 18C, and 18D, or may be a device other than terminal device 18. Each terminal device 18 may have the functions of information processing device 10, and the processor 34 of each terminal device 18 may operate the device 16 to be operated.

[0147] For example, similar to Example 2, the processor 34 of each terminal device 18 controls the volume of the speaker provided in the terminal device 18 based on the brain waves of each user. For example, by analyzing the brain waves of user B1, if it is determined that user B1 is excited, the processor 34 of each terminal device 18 controls the volume of the speaker of each terminal device 18 so that the volume of user B1 becomes smaller.

[0148] As another example, the processor 34 of each terminal device 18 may calculate the average of the brain waves of one or more users belonging to a group and determine the state of the group based on the average. For example, by analyzing the average of the brain waves of users B1 and B2 belonging to group B, if it is determined that users B1 and B2 are excited, the processor 34 of each terminal device 18 controls the volume of the speaker of each terminal device 18 so that the volumes of users B1 and B2 become smaller.

[0149] Also in Example 3, the processor 26 of the information processing device 10 may operate the device 16 based on the brain waves of each user (for example, based on the average of the brain waves). Further, the processor 26 may operate the device 16 based on the brain waves of one or more users belonging to a group (for example, based on the average of the brain waves of one or more users belonging to the group).

[0150] (Processing when the living being moves) Hereinafter, with reference to FIG. 13, the processing when the living being moves will be described. FIG. 13 shows a screen 38A.

[0151] For example, when user C1 who was participating in an online meeting stops participating in the online meeting or when the image of user C is no longer displayed in area 40C, the processor 26 of the information processing device 10 operates the device 16 based on the brain waves of the users belonging to each of groups A, B, and D.

[0152] <Example 4> Hereinafter, Example 4 will be described. In Example 4, processing for prompting a person to move is performed.

[0153] For example, the processor 26 of the information processing apparatus 10 obtains the emotional information, mental information, and psychological information of a person by analyzing the person's biological information. When a plurality of types of biological information are measured, the processor 26 may obtain the emotional information, mental information, and psychological information of the person by analyzing all or part of the plurality of biological information. The processor 26 may perform a weighting process on each biological information based on the priority order of each biological information, and obtain the emotional information, mental information, and psychological information of the person based on the result of the weighting process. In addition, a reference level is defined for various biological information, and the processor 26 may determine that biological information with the same reference level shows the same tendency in terms of emotion, mental state, and psychological information.

[0154] Based on the result of the analysis of the biological information, the processor 26 prompts the person to move to an optimal location. For example, when the processor 26 determines based on the result of the analysis of the biological information that the person is feeling uncomfortable, the processor 26 prompts the person to move to a place where the person can feel comfortable. The place where the person can feel comfortable corresponds to an example of an optimal location.

[0155] Based on the biological information measured by the biological information measuring device 12 and the environmental information measured by the environmental information measuring device 14, the processor 26 specifies an optimal location (for example, a place where a person can feel comfortable).

[0156] To explain with a specific example, when the processor 26 determines based on a person's biological information that the person feels hot, the processor 26 prompts the person to move to a cool place. The cool place corresponds to an example of an optimal place. For example, the processor 26 analyzes the person's sweating amount and brain waves, and when the sweating amount is equal to or greater than a threshold value and the brain waves indicate discomfort, the processor 26 determines that the person feels hot. For example, the environmental information measuring device 14 measures environmental information such as the temperature, humidity, and air volume of each place, and the environmental information of each place is transmitted from the environmental information measuring device 14 to the information processing device 10. The processor 26 identifies a place where the person can feel cool based on the environmental information of each place. For example, the processor 26 identifies a place where the temperature is lower than the temperature of the current place where the person is located, and prompts the person to move to the identified place.

[0157] Also, when the processor 26 determines based on a person's biological information that the person feels cold, the processor 26 prompts the person to move to a warm place. In this case, the warm place corresponds to an example of an optimal place.

[0158] For example, the processor 26 prompts each individual person to move to an optimal place. For example, as shown in FIG. 8, when there are users A, B, and C in the room 36, the processor 26 prompts each of the users A, B, and C to move to the optimal place for each user. The processor 26 may guide users with the same tendency in biological information to the same place. For example, when users A and B feel hot and user C feels cold, the processor 26 guides users A and B to a cool place and user C to a warm place.

[0159] As another example, when a person is looking at an image or text displayed on a display, if the processor 26 determines based on the person's biological information that the person feels that the image or text is difficult to see, the processor 26 prompts the person to approach the display.

[0160] As yet another example, when sound is being emitted from a speaker, if the processor 26 determines, based on a person's biometric information, that the person feels the sound is difficult to hear, the processor 26 prompts the person to approach the speaker.

[0161] For example, the processor 26 transmits information prompting movement (for example, information indicating guidance) to the terminal device 18 of the person. The processor 34 of the terminal device 18 causes the information to be displayed on the display of the UI 30. FIG. 14 shows an example of such a display. The screen 42 is a screen displayed on the display of the terminal device 18. On the screen 42, for example, a message such as "Please move to XX (a comfortable place)" and a map 44 indicating the location are displayed. On the map 44, for example, a mark 46 representing the current position of the person, a mark 48 representing the comfortable place, and an arrow 50 guiding the person to the comfortable place are displayed. The current position of the person and the comfortable place are identified by using a technology such as GPS. In the example shown in FIG. 8, information prompting movement is displayed on the displays of the respective terminal devices 18 of users A, B, and C.

[0162] FIG. 15 shows another example of a display. FIG. 15 shows a room 36. For example, the processor 26 of the information processing apparatus 10 may display guidance by light or the like on the floor of the room 36 as indicated by the reference numeral 52. For example, the processor 26 displays guidance by controlling the lighting installed in the room 36. If the place where the person is located is a road or the like, the processor 26 displays guidance on the ground or the like. In addition, the processor 26 may display guidance on a wall or may guide the person by voice. The voice may be emitted from the terminal device 18 or from another speaker. Further, the processor 26 may cause guidance to be displayed on a display installed around the person.

[0163] The processor 26 may guide a plurality of people whose biological information shows the same tendency to the same place, and guide a plurality of people whose biological information shows different tendencies to different places respectively. For example, the processor 26 may guide a plurality of people who have goodwill towards each other to the same place, and guide a plurality of people who have aversion to each other to different places.

[0164] <Example 5> Hereinafter, Example 5 will be described. In Example 5, biological information of organisms is displayed. Referring to FIG. 16, Example 5 will be described in detail. FIG. 16 shows a screen 38A. Here, as an example, it is assumed that an online meeting is being held by a plurality of users, similar to Example 3.

[0165] For example, the biological information of each user is transmitted to the terminal device 18 of each user, and the processor 34 of the terminal device 18 causes the biological information of each user to be displayed on the display. FIG. 16 shows the screen 38A of the terminal device 18A of user A, and the processor 34 of the terminal device 18A displays the biological information of each user on the screen 38A. FIG. 16 shows an example of biological information, an electroencephalogram. For example, the waveform of the electroencephalogram is displayed.

[0166] In the example shown in FIG. 16, the electroencephalogram 54 of user A1, the electroencephalogram 56 of user B1, the electroencephalogram 58 of user C1, and the electroencephalogram 60 of user D3 are displayed. For example, the processor 34 of the terminal device 18A associates the electroencephalogram 54 of user A1 with the image of user A1 and displays it within the area 40A. The same applies to the electroencephalograms of other users.

[0167] The user can select whether to display or not display the electroencephalogram. The processor of the terminal device 18A may, for example, display the electroencephalogram of each user according to the instruction of the user of the terminal device 18A, or may not display the electroencephalogram of each user, or according to the instruction of the user whose electroencephalogram is measured, display the electroencephalogram of the user, or may not display the electroencephalogram of the user.

[0168] For example, when the user of the terminal device 18A instructs to display the brain waves of users A1, B1, C1, and D3 respectively, the processor 34 of the terminal device 18A displays the brain waves of users A1, B1, C1, and D3 respectively as shown in FIG. 16.

[0169] As another example, user A1 may set the display or non-display of their own brain waves. When the display is set, the brain waves of user A1 are displayed, and when the non-display is set, the brain waves of user A1 are not displayed. The same applies to other users.

[0170] In addition, the processor 34 of the terminal device 18 may display the waveform of the brain wave and also display information indicating the emotion, mental state, or psychological state obtained by analyzing the brain wave. For example, the brain wave 58 represents a state of "awake (excited)", and the processor 34 displays a character string indicating that "awake (excited)" in the area 40C together with the brain wave 58. The same applies to other brain waves.

[0171] The processor 26 of the information processing device 10 may convert the voice data during the online meeting into text data, associate the text data with the biometric information (e.g., brain waves) of each user, and store it in the memory. In addition, the processor 26 may associate the voice data with the biometric information of each user and store it in the memory. Further, the processor 34 of the terminal device 18 may acquire the text data and display the text. For example, the voice data or text data associated with the biometric information may be searchable using the biometric information as a search key.

[0172] In addition, the processor 34 of the terminal device 18 may display brain waves representing predetermined emotions, mental states, psychological states, or intention information. For example, when brain waves not suitable for the meeting (e.g., brain waves representing drowsiness or sleep) are measured, the processor 34 displays the brain waves associated with the image of the user for whom the brain waves were measured.

[0173] Images representing emotions, mental states, psychological states, or intention information may be displayed. An example of such display is shown in FIG. 17. For example, an image 62 representing concentration, an image 64 representing drowsiness, an image 66 representing relaxation, etc. are displayed. For example, the processor 34 of the terminal device 18 associates and displays each image with the user, either together with or instead of the brain waves.

[0174] The processor 34 of the terminal device 18 may change the number, color, emphasis, and type of images, etc. according to the degree of emotion, mental state, psychological state, or intention information.

[0175] The processor 34 may associate and display a score corresponding to the degree of emotion, mental state, psychological state, or intention information with the user.

[0176] The processor 34 may process and display the user's own image (for example, an image of the user's face) or change the image of the user's background according to the degree of emotion, mental state, psychological state, or intention information. For example, when the user's mood is negative, the processor 34 displays the color of the user's face as white or bluish-white, and when the user's mood is positive, the processor 34 displays the skin color of the user's face brightly. When the user's mood is negative, the processor 34 may display the background image darkly, and when the user's mood is positive, the processor 34 may display the background image brightly.

[0177] <Example 6> Hereinafter, Example 6 will be described with reference to FIG. 18. FIG. 18 schematically shows a tomographic image of a human brain.

[0178] For example, the electroencephalogram of a person is measured by the biological information measuring device 12, and the processor 26 of the information processing device 10 identifies the activated site in the brain of the person based on the measured electroencephalogram. For example, by providing electrodes at three or more locations on a person's head to measure the electroencephalogram, the processor 26 identifies the site where the measured electroencephalogram is generated. The processor 26 may display a mark or the like indicating the identified site on the tomographic image of the person's brain, or may fill in the identified site.

[0179] Note that the tomographic image may be an image taken by a CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device, or may be an image (for example, a pre-created image) that schematically represents the brain.

[0180] <Example 7> Hereinafter, the screen on which the biological information is displayed will be described. FIG. 19 shows an example of the home screen.

[0181] The home screen 100 is displayed, for example, on the display of the terminal device 18. On the home screen 100, a "Scan" button, a "Record" button, a "Reset" button, a "File" button, and an "Edit" button are displayed. Also, the screen 100 includes display areas 102, 104, 106. In the example shown in FIG. 19, the areas 102, 104, 106 are arranged vertically in that order, but this arrangement is just an example. The arrangement may be changed by the user.

[0182] When the "Scan" button is pressed, a list of biological information measuring devices 12 that can communicate with the terminal device 18 (for example, can be paired such as BLE) is displayed on the home screen 100. When the "Record" button is pressed, the biological information measured by the biological information measuring device 12 is stored in the memory. For example, the biological information measuring device 12 and the terminal device 18 communicate, and the biological information measured by the biological information measuring device 12 is transmitted from the biological information measuring device 12 to the terminal device 18 and stored in the memory 32 of the terminal device 18. When the "Reset" button is pressed, the calibration settings are reset. When the "File" button is pressed, a screen for managing biological information files is displayed. When the "Edit" button is pressed, a setting screen (see FIG. 23) described later is displayed.

[0183] In the area 102, the biological information itself measured by the biological information measuring device 12 is displayed, and in the area 104, the information obtained by processing the biological information is displayed. Hereinafter, the unprocessed biological information will be referred to as "Raw data". For example, in the area 102, a graph of Raw data (hereinafter referred to as "Graph 1") is displayed, and in the area 104, a graph of the processed biological information (hereinafter referred to as "Graph 2") is displayed. In the area 106, the setting values of the graph displayed in the area 104 and the like are displayed.

[0184] The vertical axis (Y-axis) of Graph 1 is the potential representing the biological information (for example, [μV]), and the horizontal axis (X-axis) is time. For example, the range of the vertical axis can be changed by a pinch operation, a zoom operation, or the like.

[0185] The vertical axis (Y-axis) of the graph 2 before switching is the potential representing the biological information (for example, [μV]), and the horizontal axis (X-axis) is time. For example, the range of the vertical axis can be changed by a pinch operation, a zoom operation, or the like.

[0186] The display content of regions 104 and 106 can be switched to other display content. For example, when an operation such as a flick operation is performed to slide regions 104 and 106, the display content is switched. Of course, the display content may also be switched by another operation.

[0187] FIG. 20 shows the display content after switching.

[0188] The graph 2 before switching is, for example, a graph (such as a real-time graph) generated by performing processing set on a setting screen described later on Raw data. In region 106 before switching, setting values and the like related to the graph are displayed. For example, an index indicating that biological information (such as brain waves) is being acquired, an index indicating that the person from whom the biological information is measured is concentrated, an index indicating that the person from whom the biological information is measured is relaxed, and values of the six-axis sensor mounted on the biological information measuring device 12 (for example, values indicating the amount of movement in the axial directions of the X, Y, and Z axes, values indicating the amount of rotation around the axes of the X, Y, and Z axes, etc.) are displayed in region 106. Each value displayed in region 106 is updated, for example, at predetermined time intervals (such as every 1 second). Also, an image having a color corresponding to each value may be displayed.

[0189] The graph 2 after switching is, for example, a graph (hereinafter referred to as the "FFT graph") generated by performing FFT (Fast Fourier Transform) processing on Raw data. In the switched area 106, setting values related to graph 1, graph 2, and the FFT graph are displayed. For example, the items "Graph X", "Graph Y", "FFT Max X", and "FFT Max Y" are displayed in the area 106. The item "Graph X" is an item for setting the scale value of the X-axis common to graph 1 and the FFT graph. The item "Graph Y" is an item for separately setting the scale values of the Y-axes of graphs 1 and 2. The item "Graph1" is an item for setting the scale value of the Y-axis of graph 1, and the item "Graph2" is an item for setting the scale value of the Y-axis of graph 2. The item "FFT Max X" is an item for setting the maximum value of the X-axis of the FFT graph. The item "FFT Max Y" is an item for setting the maximum value of the Y-axis of the FFT graph.

[0190] Figure 21 shows a specific example of graph 1 of Raw data and graph 2 before switching. Figure 22 shows a specific example of graph 2 (graph after FFT processing) after switching.

[0191] Figure 23 shows an example of the setting screen. When the "Edit" button is pressed on the home screen 100, the setting screen 110 shown in Figure 23 is displayed.

[0192] The item "Micro Volts" is an item for setting the display of the Y-axes of graphs 1 and 2 to "μV".

[0193] The item "Cut Off Filter" is an item for setting the set values of filters (for example, high-pass filter (HPF) and low-pass filter (LPS)) applied to biological information. Filter processing is applied to the biological information according to the set values set here. For example, filter processing is applied to Graphs 1 and 2. The filter processing may be executed by the biological information measuring device 12, may be executed by the information processing device 10, or may be executed by the terminal device 18. By executing the filter processing, information having a specific frequency can be extracted from the measured biological information. For example, only brain waves can be extracted from biological information including brain waves and other electromyographic signals, or other electromyographic signals (for example, signals caused by eye movement, face movement, jaw movement, etc.) can be extracted. Based on the extracted signal, the device 16 may be operated. For example, the processor 26 of the information processing device 10 may distinguish and extract brain waves and other electromyographic signals, and operate the same device 16 or operate different devices 16 separately based on the brain waves and other electromyographic signals.

[0194] The item "Peak Filter" is an item for setting values related to processing of the peak value of the Raw data. For example, items for setting that measurement of biological information (for example, brain waves) is impossible when the peak value of the Raw data is equal to or greater than the set value, and items for setting set values for measuring impulse-type noise are included in the item "Peak Filter".

[0195] <Example 8> Hereinafter, Example 8 will be described. In Example 8, the processor 26 of the information processing device 10 infers the emotion, mental state, and psychological state of another person who has a specific relationship with the person based on the biological information of the person. The processor 26 may also infer the emotion, mental state, and psychological state of another person based on the emotion information, mental information, and psychological information of the person.

[0196] Specific relationships include, for example, parent-child relationships and blood relationships. For example, the processor 26 may infer the emotions, mental states, and psychological states of a child based on the biological information, emotional information, mental information, and psychological information of the parent, or may infer the emotions, mental states, and psychological states of the parent based on the biological information, emotional information, mental states, and psychological states of the child, or may infer the emotions, mental states, and psychological states of the siblings of a person based on the biological information, emotional information, mental information, and psychological information of that person.

[0197] For example, the biological information of the parent and the biological information of the child are respectively measured by the biological information measuring device 12, and when the biological information of the child is similar to the biological information of the parent (for example, when the brain wave of the child has a waveform similar to the brain wave of the parent), the processor 26 of the information processing device 10 infers that the emotions, mental states, and psychological states of the child are the same as the emotions, mental states, and psychological states of the parent. For example, when the biological information of the parent represents the emotion of "feeling good", and the biological information of the child is similar to the biological information of the parent, the processor 26 infers that the child also has the emotion of "feeling good". For example, when the child is an infant or a young child who has difficulty controlling emotions, mental states, and psychological states, the above inference is made.

[0198] As another example, when the parent is a person who has difficulty controlling emotions, mental states, and psychological states (for example, a person suffering from dementia, etc.), the processor 26 may infer the emotions, mental states, and psychological states of the parent based on the biological information, emotional information, mental information, and psychological information of the child.

[0199] As yet another example, the processor 26 may infer the emotions, mental states, and psychological states of people with similar living habits. For example, among people who have lived together for a long time, their living habits may become similar. In that case, the processor 26 infers the emotions, mental states, and psychological states of another person with living habits similar to that person based on the biological information, emotional information, mental information, and psychological information of that person.

[0200] <Example 9> Hereinafter, Example 9 will be described. In Example 9, the processor 26 of the information processing apparatus 10 notifies a target person, who is a human, of specific information that the target person cannot know. For example, the processor 26 transmits the specific information to the terminal device 18 of the target person.

[0201] The specific information notified to the target person is, for example, visual information, environmental information, and information regarding the atmosphere related to a specific location. For example, information that cannot be known unless the target person is at the specific location is the information notified to the target person. The location is designated, for example, by the administrator of the information processing apparatus 10, the target person, or the like. Further, information indicating a danger that has occurred or may occur at the specific location may be transmitted to the terminal device 18 of the target person according to the designation by the administrator, the target person, or the like.

[0202] The visual information is, for example, an image captured by a fixed-point camera installed at a specific location, an image captured by a camera worn by a person at a specific location, or the like. The visual information may include sound information measured by a microphone installed at a specific location or a microphone worn by a person. The environmental information is information measured by various sensors installed at a specific location (for example, temperature, humidity, air chamber, weather, etc.), information measured by sensors worn by a person at a specific location, or the like. The information regarding the atmosphere is, for example, information indicating that a specific location is a comfortable or uncomfortable place for a person, information indicating that a specific location is a place where a person can concentrate or be distracted, or the like. For example, when a person at a specific location inputs information regarding the atmosphere of the specific location to their own terminal device 18, the input information is transmitted from the terminal device 18 to the information processing apparatus 10.

[0203] For example, when the subject uses the terminal device 18 to specify a specific location and requests the acquisition of specific information at that specific location, the information indicating the request is transmitted from the terminal device 18 to the information processing device 10. The processor 26 of the information processing device 10 transmits the specific information at the specified specific location to the terminal device 18 in response to the request. The processor 34 of the terminal device 18 causes the specific information to be displayed on the display or generated as sound from the speaker.

[0204] In addition, the processor 26 of the information processing device 10 observes the situation of each location, and when the situation at the specific location specified by the subject meets the predetermined conditions, the processor 26 may transmit the specific information at the specific location to the terminal device 18 of the subject. The situation of the location is, for example, the situation related to the environment obtained by analyzing environmental information (such as temperature, humidity, weather, etc.), the situation obtained from visual information (such as the congestion situation of streets, trains, etc.), and the dangerous situation obtained from visual information, environmental information, etc. The situation of the location may be inferred or predicted from visual information, environmental information, etc.

[0205] For example, when the weather at the specific location specified by the subject meets the predetermined weather or is predicted to become the predetermined weather, the processor 26 of the information processing device 10 transmits the information related to the weather at the specific location to the terminal device 18 of the subject. Also, when the congestion situation at the specific location becomes or is predicted to become the predetermined congestion situation, the processor 26 of the information processing device 10 transmits the information related to the congestion situation at the specific location to the terminal device 18 of the subject.

[0206] In addition, the processor 26 of the information processing device 10 may select the specific information transmitted to the terminal device 18 of the subject based on the purpose of the subject (such as avoiding a crisis, etc.). The purpose is specified by the subject, for example.

[0207] For example, when a vehicle is running wild, if a person uses the terminal device 18 to give an instruction to send information about the vehicle running wild to the information processing device 10, information about the vehicle (for example, information including information indicating danger) is sent from the terminal device 18 to the information processing device 10. The information about the vehicle includes the position information of the terminal device 18. Also, when the vehicle running wild is photographed by a camera worn by a person, the photographed image may be included in the information about the vehicle. When the processor 26 of the information processing device 10 receives the information about the vehicle, it acquires an image (for example, an image taken from a higher position) taken by a fixed-point camera installed around the position of the terminal device 18 from the fixed-point camera, and sends the image to the terminal device 18 of the target person. The image is displayed on the display of the terminal device 18 of the target person. Also, the processor 26 may send information indicating danger to the terminal device 18 of the target person. Information indicating the danger is displayed on the display of the terminal device 18 of the target person. Also, when a plurality of fixed-point cameras are installed, the processor 26 may send an image taken by the fixed-point camera most effective for danger detection to the terminal device 18 of the target person.

[0208] Also, the processor 26 may select information included in specific information, or process specific information, according to the attributes of the target person (such as age, gender, medical history, etc.), and send the specific information to the terminal device 18 of the target person. For example, since differences in visual function can occur depending on age (for example, due to cataracts, glaucoma, etc., differences in visual function can occur), the processor 26 may adjust the sharpness of the image included in specific information according to the age of the target person. Also, since differences in crisis awareness can occur depending on gender and age, the processor 26 may select information included in specific information according to the gender and age of the target person. For example, the processor 26 may include only an image in specific information, or include a character string (for example, a character string indicating danger) in specific information, or include both an image and a character string in specific information, according to the gender and age of the target person.

[0209] <Example 10> Hereinafter, Example 10 will be described.

[0210] For example, the biological information of a plant may be measured by the biological information measuring device 12, and the device 16 may be operated based on the measured biological information. The biological information to be measured is, for example, the potential of the plant (including potentials such as membrane potential, etc.) or the moisture content, etc. Of course, information other than these may be measured as biological information.

[0211] For example, when the biological information of a plant is being measured, if the growth of the plant is better when music is played near the plant compared to when no music is played, the processor 26 of the information processing device 10 operates the device 16 such that music flows from the device 16, such as a music player, so that the biological information (for example, potential) measured when the music is being played is continuously measured from the plant. Also, the processor 26 may control how water and fertilizer are given to the plant. For example, the processor 26 may give water and fertilizer to the plant so that the biological information is continuously measured from the plant.

[0212] In addition, the processor 26 may detect a change in the state of the plant based on the biological information measured from the plant, and display information (such as an image or a character string, etc.) corresponding to the change on the display of a device such as the terminal device 18. The processor 26 may anthropomorphize the plant and display information corresponding to the change on the display.

[0213] The processor 26 may determine whether the plant has a disease based on the biological information measured from the plant, and display the result of the determination on the display of a device such as the terminal device 18.

[0214] In addition, the moisture content of a living organism changes over time, and this change is reflected in the bioelectric potential. The bioelectric potential is measured by the biological information measurement device 12, and the processor 26 of the information processing device 10 may detect the behavior and abnormalities of the living organism based on the measured bioelectric potential. For example, when an abnormality is detected, the processor 26 may transmit warning information indicating the abnormality to a device such as the terminal device 18. For example, when the bioelectric potential of a person is measured and an abnormality of the person is detected based on the bioelectric potential, warning information is transmitted to the terminal device 18 of the person, and the warning information is displayed on the display of the terminal device 18.

[0215] Also, generally, in the morning, moisture is evenly distributed throughout a person's body, and during the day, as the person is active, moisture gradually increases in the lower part of the body. This change in the moisture content is reflected in the bioelectric potential. The processor 26 of the information processing device 10 may measure the bioelectric potential of each part of a person and measure time based on the change in the bioelectric potential. For example, the change in the bioelectric potential may be used as a clock or a timer.

[0216] In addition, the processor 26 of the information processing device 10 may measure the biological information (such as brain waves, etc.) of animals other than humans and infer the state of the animal (such as illness) and the environment in which the animal is placed.

[0217] In addition, the processor 26 of the information processing device 10 may detect the type of living organism wearing the biological information measurement device 12.

[0218] For example, the skull sizes of humans and animals other than humans are different. When the biological information measurement device 12 has a band for attaching the biological information measurement device 12 to the head, and the biological information measurement device 12 is attached to the head by winding the band around the head, the processor 26 may detect the length of the band by a sensor or the like, and detect the type of animal to which the biological information measurement device 12 is attached based on the length of the band. To explain with a specific example, since the skull sizes of humans and dogs are different, the lengths of the bands accordingly differ. The processor 26 detects the length of the band and detects whether the organism to which the biological information measurement device 12 is attached is a human or a dog.

[0219] As another example, when the biological information measurement device 12 is a device attached to the ear of an organism, the processor 26 may detect the length of the ear hair of the ear to which the biological information measurement device 12 is attached by a sensor or the like, and detect the type of animal to which the biological information measurement device 12 is attached based on the length of the hair.

[0220] As yet another example, since body temperatures differ depending on the type of organism, the processor 26 may measure the body temperature of the organism wearing the biological information measurement device 12, and identify the type of the organism based on the body temperature.

[0221] As yet another example, since height and weight differ depending on the type of organism, and the resistance value accordingly differs, the processor 26 may identify the type of organism wearing the biological information measurement device 12 based on the resistance value.

[0222] The type of the organism (for example, human, dog, plant, etc.) may be associated with the biological information measured from the organism. The association may be performed by the biological information measurement device 12 or by the information processing device 10. By doing so, it is possible to identify from which organism a certain biological information was measured. As described above, when the type of the organism from which the biological information is measured is identified, information indicating the identified type may be associated with the biological information.

[0223] Biological information such as brain waves may be used for communication between animals. For example, communication may be performed using biological information between humans, or between a human and an animal other than a human (e.g., a dog). When the type of organism for which the biological information is measured is specified as described above, the processor 26 may change the method of converting brain waves in the communication, etc.

[0224] <Example 11> Hereinafter, Example 11 will be described. In Example 11, a stimulus is applied to an organism, and the processor 26 of the information processing apparatus 10 associates the stimulus information indicating the content of the stimulus with the biological information measured from the organism. The associated stimulus information and biological information are stored, for example, in the memory 24 of the information processing apparatus 10 or the like. A plurality of pieces of biological information may be associated. For example, brain waves, pulse, blood pressure, facial color, body temperature, etc. may be associated with the stimulus information.

[0225] The stimulus is, for example, an image such as a still image or a moving image, an environment, a taste, etc. For example, an image expressing joy, anger, sorrow, and pleasure, an image or environment that makes one feel fear, an image or environment that makes one feel relaxation, etc. are given as stimuli. The image is, for example, a virtual image such as VR (Virtual Reality). Of course, an image representing a real object may be used. The stimulus may be an electrical stimulus applied by an electrode provided in the biological information measuring device 12.

[0226] For example, when a person is looking at an image, biological information (e.g., brain waves) is measured from the person by the biological information measuring device 12, and the image and the biological information are associated. Note that the measured brain waves are brain waves emitted from the left brain, brain waves emitted from the right brain, or brain waves emitted from both the left and right brains.

[0227] In addition, the processor 26 may associate the words (i.e., voice information) uttered by a person when the stimulus is applied with the stimulus information and the biological information. Furthermore, the processor 26 may associate the information indicating the movement of the person with the stimulus information and the biological information.

[0228] Virtual images such as VR images may be images representing backgrounds photographed in people's daily lives, and ambient sounds may be reproduced together with the VR images.

[0229] Also, as the VR image, an image corresponding to what the person whose biological information has been measured has experienced so far may be used. The things that the person has experienced so far may be investigated in advance (for example, through a questionnaire).

[0230] When the biological information measured from the person viewing the VR image changes significantly, the processor 26 may associate the changed part of the biological information with the stimulus information as valid biological information. For example, when the amount of change in the biological information is equal to or greater than a threshold value, the processor 26 associates the biological information of that part with the stimulus information as valid biological information.

[0231] The order in which the stimulus information is given to a person may be controlled. For example, after giving the person visual information that the person dislikes, auditory information, olfactory information, or somatic sensory information that the person likes may be given to the person. The processor 26 may associate the biological information measured at that time with each stimulus information.

[0232] The processor 26 may change the stimulus information given to a person according to the change in the measured biological information.

[0233] Also, the processor 26 may change the playback speed of the VR image, change the viewpoint of the VR image, change the color tone of the VR image, or change the image quality or focus of the VR image.

[0234] The associated stimulus information and biological information may be utilized in digital marketing (for example, video production, attractions, products such as devices, etc.) and digital healthcare (for example, services that record daily life and evaluate life from the ratio of stress and refreshment levels, etc.).

[0235] <Example 12> The biological information measurement device 12 may be shared by a plurality of people or the like. For example, after person A uses the biological information measurement device 12, another person, person B, may use the biological information measurement device 12 that person A was using.

[0236] For example, a contact-type biological information measurement device 12 (that is, a device that measures biological information by electrodes or sensors contacting a living organism such as a person) may output information indicating that the living organism (for example, a person) wearing the biological information measurement device 12 has changed, or information prompting cleaning. Outputting the information may be, for example, displaying the information on a display, emitting the information as sound from a speaker, or both.

[0237] For example, when a microphone is provided in the biological information measurement device 12, the biological information measurement device 12 determines whether the person using the biological information measurement device 12 has changed by recognizing the person using the biological information measurement device 12 based on the voice of the person input from the microphone. The biological information measurement device 12 may recognize the person based on an image generated by photographing the person using the biological information measurement device 12 with a camera provided in the terminal device 18 or the like.

[0238] As another example, the account of the person using the biological information measurement device 12 is managed by the information processing device 10, the biological information measurement device 12, the terminal device 18, the server, or the like. When the account is switched to another account, the biological information measurement device 12 may determine that the person using the biological information measurement device 12 has changed.

[0239] By outputting information indicating that the person using the biological information measurement device 12 has changed, the person using the biological information measurement device 12 can recognize that the biological information measurement device 12 has been used by another person. Thereby, for example, the person newly using the biological information measurement device 12 can recognize that the biological information measurement device 12 needs to be cleaned. The same applies when information prompting cleaning is output.

[0240] Further, a sensor or the like for detecting that the biological information measuring device 12 has been cleaned may be used. For example, a sensor for detecting that electrodes or sensors for measuring biological information have been cleaned with alcohol (for example, a sensor for detecting alcohol) is used. When the biological information measuring device 12 is about to be used in a state where the electrodes or sensors have not been cleaned with alcohol (for example, when the power of the biological information measuring device 12 is turned on), the biological information measuring device 12 outputs information indicating that the biological information measuring device 12 has not been cleaned. Further, the biological information measuring device 12 does not necessarily need to turn on the power of the biological information measuring device 12. When the biological information measuring device 12 is about to be used in a state where the electrodes or sensors have been cleaned with alcohol, the biological information measuring device 12 may output information indicating that the biological information measuring device 12 has been cleaned, or may not output such information.

[0241] <Example 13> Hereinafter, Example 13 will be described with reference to FIG. 24. FIG. 24 shows a screen 38A. The screen 38A is the same screen as the screen according to Example 3 (see FIG. 12).

[0242] In Example 13, information representing emotions, intentions, mental states, psychological states, etc. (for example, an image or numerical value representing a bar) is displayed. Emotions, intentions, mental states, and psychological states are estimated by the processor 26 based on the biological information of the user. For example, a bar representing the degree of concentration, relaxation, stress, etc. of the user is displayed, or a numerical value representing them is displayed. Bars or numerical values for all users may be displayed, or bars or numerical values for users designated by an administrator or the like may be displayed.

[0243] In the example shown in FIG. 24, bars and numerical values representing the mental state and the like of user C1 are displayed. Bar 120 is an image representing the degree of relaxation of user C1. Bar 122 is an image representing the degree of stress of user C1. The degree of relaxation is "100", and the degree of stress is "30". These numerical values are calculated by processor 26 based on the biological information of user C1.

[0244] In addition, processor 26 may identify an event that contributed to the state of the user (such as emotion, will, mental state, psychological state, etc.), and calculate the degree to which the event contributed to the state of the user. For example, processor 26 may identify an event that contributed to stress or recovery from the stress, and display information indicating the degree of the contribution on screen 38A. By doing so, it becomes easy for the user to understand what kind of event affected the user's state and to what extent. For example, processor 26 detects the occurrence of an event by various sensors and the like, and estimates the change in the user's state when the event occurred and the user's state before and after the occurrence of the event based on the biological information of the user. Processor 26 estimates, for example, the degree to which the event contributed to the user's state based on the change in the user's state.

[0245] Further, the processor 26 may store in the memory 24 history information indicating a history regarding the occurrence of an event, the user's biological information before and after the occurrence of the event, changes in the user's biological information, the user's state before and after the occurrence of the event, and changes in the user's state. The processor 26 may reproduce past events based on the history indicated by the history information. For example, the processor 26 may record events before and after the time when the user relaxes and reproduce the same situation as that time. For example, a target value for relaxation is specified, and the processor 26 reproduces the situation in which the target value is obtained. The processor 26 may reproduce the situation by combining music playback, display of images, etc., and operation instructions for people. The processor 26 may acquire the state of the user's brain waves, classify the state of the brain waves, and record a history associated with working hours (for example, time excluding sleeping hours). Note that the processor 26 may display a numerical value or bar indicating a state such as relaxation so as to be sandwiched between images of a meeting by a virtual camera.

[0246] <Example 14> Hereinafter, Example 14 will be described with reference to FIGS. 25 and 26. As in FIG. 17, FIG. 25 shows images representing each of users A, B, and C. Further, information indicating the state of each user (for example, brain waves, etc.) is displayed in association with each user. Here, as an example, it is assumed that users A, B, and C are participating in the same online meeting.

[0247] For example, when all users participating in an online meeting or a number of users equal to or more than a predetermined number have the same state (for example, the same emotion, intention, mental state, or psychological state), the processor 26 may display the frame of the image of the users having the same state in a specific color (for example, red, etc.) or apply a specific decoration. The state of the user is estimated based on the user's biological information.

[0248] For example, when users A, B, and C have the same state (e.g., awake state), as shown in FIG. 26, the processor 26 displays the entire frame surrounding the images of users A, B, and C in a specific color or applies a specific decoration (e.g., a thick frame). By doing so, it can be shown that users A, B, and C have the same state. For example, when the states of all users participating in an online meeting are in the awake state, such display and decoration indicate that the meeting mode is a passionate mode.

[0249] As in Example 14, information collectively representing the states of a plurality of users participating in a meeting may be displayed.

[0250] <Example 15> Hereinafter, Example 15 will be described.

[0251] The data transmitted from the biological information measuring device 12 to another device (e.g., the information processing device 10) may be data generated by applying processes such as FFT instead of the raw data of the biological information. By doing so, it is possible to suppress the leakage of information that is included in the raw data but not included in the data after processes such as FFT.

[0252] Also, information indicating the items of the measured biological information (e.g., information indicating channels or parts) may be encrypted by the biological information measuring device 12 and transmitted from the biological information measuring device 12 to another device (e.g., the information processing device 10). By doing so, it becomes difficult to know what items of biological information the transmitted biological information is, so the security of the biological information is improved.

[0253] Also, when lending or selling the measured biological information or information obtained from the biological information to the user, the processor 26 may restrict the use of the biological information. The content of the restriction is, for example, deleting the information when a predetermined period has elapsed, or returning the information to the provider. Also, the place where the use of the information is permitted and the purpose of the use may be determined.

[0254] <Example 16> Hereinafter, Example 16 will be described.

[0255] In Example 16, an AI assistant (i.e., an artificial intelligence assistant) that emits voice is used. For example, the AI assistant executes processing or operates a device according to the user's biological information or the user's instructions. The AI assistant may be installed, for example, in the user's terminal device, in a device other than the terminal device, or in the information processing apparatus 10, a server, or the like.

[0256] For example, the processor 26 may change the volume of the voice emitted from the AI assistant or the speaking speed of the AI assistant according to the user's biological information.

[0257] Specifically, when the user's biological information indicates that the user wants to relax, the processor 26 changes the voice emitted from the AI assistant to a voice preferred by the user (for example, high pitch, low pitch, speaking speed, etc.) or changes it to the voice of the user's family member.

[0258] Also, when the user's biological information indicates that the user is in a hurry, the processor 26 increases the speaking speed of the AI assistant or causes the AI assistant to output voice quickly.

[0259] Also, when the user learns from the voice emitted by the AI assistant, the processor 26 may control the AI assistant to emit voice at a conversation speed and volume that are highly effective for learning for the user. The learning effect of the user is estimated based on the user's biological information. For example, the processor 26 changes the conversation speed and volume of the AI assistant based on the user's biological information.

[0260] Also, when the user is participating in an online meeting, a conference call, etc., the processor 26 estimates the state of the user's counterpart based on the biometric information of the counterpart, and based on the result of the estimation, changes the voice uttered by the user in a manner in which the voice is likely to be transmitted to the counterpart. For example, when the user is angry and utters a voice that makes the counterpart scared, the processor 26 may change the user's voice to a mild voice, or may express the content uttered by the user by means of graphics, text, etc. without transmitting the voice to the counterpart. Such processing may be executed by an AI assistant.

[0261] Also, in an online meeting or the like, there may be a case where a certain user (for example, the first user) wears the biometric information measurement device 12 and the other user (for example, the second user) who is the counterpart does not wear the biometric information measurement device 12. In this case, the biometric information of the first user is acquired, and the biometric information of the second user is not acquired. Even in such a case, if the first user can recognize the expression of the second user through the screen, the biometric information of the first user may change due to a change in the expression, voice, etc. of the second user. When such a change occurs in the biometric information of the first user, the processor 26 determines that the change in the biometric information of the first user has occurred due to the second user. Based on that determination, the processor 26 may estimate the state (for example, emotion, mental state, psychological state) of the second user. For example, when the expression on the face of the second user looks painful, the first user who sees that face may have a feeling of apology, so the processor 26 infers the state of the second user based on the biometric information of the first user. Also, the AI assistant may inform the first user how to interact with the second user who is in such a state.

[0262] In addition, when adding voices to animation characters or adding dubbed voices to dramas or movies, the user who has the authority to determine the voice can listen to the voice of the voice actor temporarily assigned to the character or actor, and the processor 26 may determine the official voice actor based on the biometric information of the user. For example, when biometric information indicating approval is obtained from the user, the processor 26 determines the voice actor who uttered the voice as the official voice actor. When biometric information indicating discomfort or uneasiness is obtained from the user, the processor 26 may guide the user to another voice actor.

[0263] <Example 17> Hereinafter, Example 17 will be described.

[0264] This embodiment may be applied to content services such as games. For example, the game may be provided to the user via the user's terminal device, or may be provided to the user via other devices (such as game devices). The game may be an online game, or may be a game provided by executing a program installed in the terminal device or game device. Further, a recording medium storing the game program may be used, and the game may be provided by the terminal device or game device executing the program.

[0265] For example, the biometric information of the user when the user is playing a game is measured by the biometric information measuring device 12. The processor 26 may change the progress of the game, the scene of the game, the voice of the game, the production of the image, etc. according to the emotion of the user obtained from the biometric information.

[0266] For example, for users who are not good at scary games or scary things, when biometric information indicating an emotion of being scarier than a predetermined level is measured, the processor 26 may omit violent scenes, lower the voice of the game, change the production to a simpler one, or change the expression to an easier expression.

[0267] On the other hand, for users who prefer more terrifying scenes, if biometric information indicating a feeling of being scarier than a predetermined level is not measured, the processor 26 may change the game presentation so that more violent scenes or scenes with more emphasized terrifying effects appear.

[0268] Note that the user's preferences are set before starting the game. Also, the processor 26 may change the game presentation in conjunction with the biometric information according to the user's medical conditions and age-related loads. Further, the processor 26 may change the presentation for educational reasons.

[0269] The processor 26 may determine the user's state based on the user's biometric information at predetermined intervals. Thereby, the scene can be changed in real time, and the user can proceed with the game without having discomfort or a sense of incongruity. Note that scenes that cannot be changed due to the game story may be set.

[0270] Also, when a feeling that the game is difficult or a feeling that the game is easy is detected, the processor 26 may change the game difficulty according to that feeling. For example, the processor 26 may increase or decrease the action level of the AI that becomes the user's enemy, increase or decrease the number of enemy characters, or increase or decrease the number of hints to reach the answer when a reasoning game is being executed. Further, the processor 26 may increase or decrease setting values such as the attack power and defense power of the character operated by the user.

[0271] The processor 26 may notify the user of the change in difficulty based on the biometric information. By doing so, the user can grasp, for example, at what difficulty settings the game could be cleared. In this case, the game play time, the change in the user's feelings during that play time, the game difficulty, etc. may be displayed by a graph or other figure.

[0272] Also, when the user obtains the desired state by playing the game, the information indicating that state may be stored in the memory 24 of the information processing apparatus 10 as history information. The processor 26 may change the game scene or difficulty level so that the user's state becomes the desired state. For example, when the target state of relieving stress and having a refreshed mood is set in the information processing apparatus 10, the processor 26 may change the game scene or difficulty level so that the user's state becomes that target state. For example, the processor 26 may change the game scene to a scene representing a clear blue sky and a plain. As another example, the processor 26 may change the enemy character to a weak character so that the user's character can easily defeat the enemy character.

[0273] <Example 18> Hereinafter, Example 18 will be described.

[0274] In Example 18, the biological information measurement device 12 is a wearable device and is worn on the user's ear. The wearable device may have a voice assistant function. The voice assistant function is, for example, a function of assisting the user based on the user's biological information. The voice assistant function may be realized by AI.

[0275] Hereinafter, the processing according to Example 18 will be described.

[0276] First, a wearable device, which is an example of the biological information measurement device 12, measures biological information (for example, electrocardiogram waveform, electroencephalogram, electromyogram waveform, skin potential, etc.) from the user's ear (for example, the external auditory canal), and further measures the user's voice. The wearable device associates and stores the biological information and the voice information in chronological order. These pieces of information may be stored in the wearable device, may be stored in the user's terminal device, or may be stored in the information processing apparatus 10. Here, as an example, it is assumed that these pieces of information are stored in the information processing apparatus 10.

[0277] The processor 26 of the information processing apparatus 10 determines whether there is a large fluctuation in the user's biometric information. For example, when the biometric information is an electroencephalogram, a large fluctuation refers to an epileptic abnormal wave. When the biometric information is an electrocardiogram waveform, a large fluctuation refers to a waveform indicating an abnormality of the myocardium such as arrhythmia.

[0278] When the processor 26 detects that a large fluctuation has occurred in the user's biometric information, it records the time when the large fluctuation occurred, the location where the user was at that time, and information indicating the user's state (hereinafter, these information are referred to as "context information"). The processor 26 may remove noise and determine whether the large fluctuation is due to a psychosomatic stimulus response.

[0279] When the above large fluctuation (that is, a change in the psychosomatic stimulus response) is a critical situation, the processor 26 controls the voice function of the wearable device. A critical situation is, for example, when a heart disease has occurred (for example, when an arrhythmia is detected based on the pulse, which is biometric information), when the user has fallen (for example, when the user's fall is detected based on the electromyogram waveform or the data of the acceleration sensor, which is biometric information), when the user has swallowed (for example, when the electromyogram waveform, which is biometric information, contains irregular waveforms, and it is detected that there is a possibility that swallowing has occurred based on the irregular waveforms), and so on.

[0280] In addition, the processor 26 transmits the above context information, biometric information, and voice information to a device (such as a PC) installed at a predetermined location (such as a hospital). At this time, the processor 26 transmits the context information, biometric information, and voice information measured during a time period (for example, a time period of a predetermined length) before and after including the time point when the large fluctuation occurred to the device.

[0281] In addition, the processor 26 may control the voice emitted from the wearable device based on the above context information, biological information, and voice information. For example, the processor 26 controls the playback speed of the content played on the wearable device. Specifically, the processor 26 changes the reading speed of learning videos or e-books. In addition, the processor 26 may stop the playback of the content. Further, the processor 26 may play content (such as subjects, comics, music, etc.) that suits the user's preferences. Also, the processor 26 may execute an auto-call function. The auto-call function here is a function that automatically notifies a pre-specified emergency contact (such as family members or a regular doctor, etc.) by phone or message. Multiple people may be notified by phone or message simultaneously, or the external person may be gradually notified by phone or message. Also, the processor 26 may change the contact based on the severity of the user's medical condition. Of course, the user himself / herself may be notified.

[0282] The above voice assistant function may be switched for each wearable device or for each user account. Also, a function may be provided to set whether to execute the voice assistant function based on biological information. The voice assistant function of the wearable device may be executed without relying on biological information. Also, the scenarios where the voice assistant function based on biological information is applied may be limited to the scenarios of diagnosing signs of illness or may be limited to the reading function such as content services. A function for setting such limitations may be provided.

[0283] The processor 26 may let the AI that realizes the voice assistant function learn the above context information, biological information, and voice information.

[0284] For example, the processor 26 may accumulate voice information when the user is speaking in a relaxed state and let the AI learn the voice information. For example, by applying technologies such as voice synthesis and voice adjustment, the AI may synthesize a voice assistant voice optimized for biological reactions and read out the voice.

[0285] As another example, the processor 26 may accumulate voice information when the user is speaking in a stressed state and let the AI learn the voice information. For example, by applying voice synthesis and voice adjustment technologies, the AI may synthesize an optimized voice assistant voice for the user to regain calmness and read out the voice.

[0286] As yet another example, the processor 26 may accumulate voice information when the user is nervous and let the AI learn when, where, and in what state the user became nervous. For example, by applying voice synthesis and voice adjustment technologies, the AI may synthesize an optimized voice assistant voice for the user to regain calmness and read out the voice.

[0287] As yet another example, the processor 26 may convert the user's voice into text, accumulate the text-converted data in chronological order together with information such as information indicating the user's emotions, anger, sorrow, joy, and let the AI learn these. It may also be stored in the memory. Note that the emotions of anger, sorrow, joy, and pleasure may be selected by the user. Further, the processor 26 may change the size, color, etc. of the text-converted characters according to the user's voice. By applying voice synthesis and voice adjustment technologies, the AI may synthesize an optimized voice assistant voice for the user to regain calmness and read out the voice.

[0288] For example, the processor 26 associates and records the user's voice information obtained from the user's terminal device (such as a smartphone or a PC), information such as icons, characters, and pictures input by the user, and the biometric information measured from the user. The processor 26 converts the user's biometric information into emotional index data such as emotions of anger, sorrow, joy, pleasure, stress, concentration, relaxation, load, understanding, proficiency, or interest.

[0289] For example, as a parameter indicating the mental activity state, electro dermal activity may be used. Also, skin potential level (SPL), which is the DC component of skin potential activity (SPA), may be used. Generally, SPL shows a high negative value when the arousal level is high (i.e., when excited), and a high positive value when drowsy or relaxed. Skin potential reflex (SPR), which is the AC component of skin potential activity (SPA), may be used. SPR frequently occurs due to stimuli caused by external environmental changes such as pain, touch, hearing, and vision, as well as deep breathing, body movements, thoughts, etc.

[0290] The processor 26 visualizes and displays the degree and type of the state (e.g., joy, anger, sorrow, stress, concentration, relaxation, load, understanding, proficiency, interest, etc.) represented by the emotion index data, along with the information indicating the user's state acquired from interfaces such as cameras and keyboards. For example, the processor 26 may change the display color, the type of font of the characters, or the size of the characters according to the degree and type of the state. For example, in the reading function realized by existing software, the processor 26 may change the speaking style (e.g., joy, anger, sadness, fear, kindness, etc.) and the tone of voice (e.g., emphasis, speaking speed, voice height, intonation, volume, etc.) according to the stimulus response (e.g., relaxation, stress, tension, etc.). The processor 26 may also change the volume and speed of the voice of the reading function according to the degree of the user's emotion.

[0291] Each of the above examples may be applied to, for example, e-books, movie contents (including contents distributed via the Internet), e-learning materials, online meeting tools, etc.

[0292] As yet another example, when creating a document or material, if a function to change the size and color of characters and figures according to the user's biometric information is executed, if this function is always executed, the unity of the document or material may be lost. Therefore, in specific areas of the document or material, the processor 26 may not change the size and color of characters and figures according to the biometric information. In other words, the processor 26 may change the size and color of characters and figures within the permitted change area according to the user's biometric information.

[0293] Also, when the reading function is executed and the characters described in the document or material are read aloud, the processor 26 may change the volume and speed of the voice during the reading according to the user's biometric information. Also, the reading speed may be limited to a range that is neither too fast nor too slow.

[0294] Note that the time required for the process of determining the emotion, mental state, or psychological state based on biometric information, and the time required for measuring the emotion, mental state, or psychological state may vary depending on the type of emotion, mental state, or psychological state. For example, the measurement interval for concentration and relaxation is 1 second, the measurement interval for interest, understanding, proficiency, activity load, emotional intensity, and creativity is 1 to 5 seconds, and the measurement interval for pleasure and discomfort is 5 seconds or 10 seconds. Of course, these times are just examples and may change depending on the performance of the biometric information measurement device 12 and the like.

[0295] When there is such a variation in time as described above, in a specific mode, the processor 26 may determine only some of the predetermined emotions, mental states, or psychological states without determining all of the predetermined emotions, mental states, or psychological states, or may limit the time and scene for making the determination. For example, when the user is studying, the processor 26 may determine only the learning effect based on the user's biometric information and may not determine emotions other than the learning effect. However, when the user's state is a specific state (for example, when the user has a disease such as a basic disease), the processor 26 may always determine all of the predetermined emotions, mental states, or psychological states.

[0296] In addition, when the user stops wearing the wearable device but is wearing another biological information measurement device 12, the processor 26 may determine the emotion, mental state, or psychological state based on the biological information measured by the other biological information measurement device 12. For example, when the user is wearing both a wearable device and a smartwatch and stops wearing the wearable device, the emotion, mental state, or psychological state may be determined based on the biological information measured by the smartwatch.

[0297] <Example 19> Hereinafter, Example 19 will be described. In each of the above-described examples, when biological information in a specific frequency band (for example, brain waves in a specific frequency band) is measured and biological information in other frequency bands outside the specific frequency band is not measured, the processor 26 may estimate the biological information in the other frequency bands. For example, the processor 26 estimates biological information that is of the same type as the measured biological information but in a frequency band different from the measured frequency band.

[0298] For example, when only the potential in the frequency band of alpha waves, which is an example of brain waves, is measured and the potential in frequency bands other than alpha waves is not measured, the processor 26 uses information measured by various sensors (for example, a gyro sensor or an acceleration sensor, etc.) at the time of measurement, environmental information (for example, information indicating temperature, humidity, weather, location, etc.), and information regarding the user's current behavior (for example, Based on one or more pieces of information among information indicating activities such as during exercise, during desk work, during sleep, etc., information regarding the user's current illness, and information regarding the measurement history of the measured alpha waves (for example, information indicating the shape, period, amplitude size, etc. of the alpha wave waveform), estimate brain waves in frequency bands that have not actually been measured (for example, brain waves other than alpha waves such as beta waves) or brain waves in frequency bands that have not been clearly detected. For example, the processor 26 estimates brain waves in unmeasured frequency bands based on the correlation relationships of the above various pieces of information. The processor 26 may output the estimated brain waves in that frequency band to a terminal device or the like. For example, the estimated brain waves in the frequency band are displayed on the display of the terminal device. At this time, the measured brain waves in the frequency band (for example, alpha waves) and the estimated brain waves in the frequency band (for example, beta waves) may be displayed as a graph having the same scale (for example, a graph with the horizontal axis indicating time and the vertical axis indicating potential). The processor 26 may output the estimated brain waves in that frequency band as candidates for brain waves in that frequency band. Even when alpha waves are not measured and only beta waves are measured, alpha waves may be estimated in the same manner. It is also acceptable.

[0299] When biometric information in a plurality of frequency bands has been measured, the processor 26 may estimate biometric information in a frequency band that has not been measured based on the biometric information in the plurality of frequency bands. For example, when alpha waves and beta waves are measured and no other brain waves are measured, the processor 26 may estimate brain waves in other frequency bands based on the actually measured alpha waves and beta waves.

[0300] In addition, when a plurality of candidates are estimated, the processor 26 may display the plurality of candidates on the display of the terminal device or output a voice indicating that a plurality of candidates have been estimated. Further, the processor 26 may change the display order of the plurality of candidates according to the probability or likelihood of occurrence of each of the plurality of candidates, or display a score corresponding to the magnitude.

[0301] For example, when measuring brain waves using a wearable device, it may be difficult to detect bioelectrical potentials generated from a location relatively far from the ear on the head. In such a case, it is conceivable to estimate brain waves in a frequency band that has not been measured based on the measured brain waves in a frequency band and other information (for example, environmental information or information about the user).

[0302] The on / off of the function of estimating biometric information in a frequency band that has not been measured may be set by the user. For example, when the user turns on the function using a terminal device or the like, the function is executed. When the user turns off the function using a terminal device or the like, the function is not executed. Since the unmeasured biometric information (for example, brain waves in a frequency band that has not been measured) may not actually be generated from the user's brain, it is conceivable to turn off the function in such a case.

[0303] <Example 20> Hereinafter, Example 20 will be described.

[0304] As the BMI, when using brain waves or the like for communication transmission between people, identify the information to be transmitted and provide a means for transmitting the information to be transmitted.

[0305] For example, the extraction of the transmission target and the identification of the information content to be conveyed are performed, and the information transmitting side conveys them to the receiving side. The transmission target may be one receiving side for one transmitting side, or multiple receiving sides (for example, a large number of receiving sides) for one transmitting side. The information content to be conveyed is information related to people's thoughts and intentions or status information, such as approval, consent, opposition, rejection, hesitation, confusion, etc. When information is provided to a person, the information transmitting side conveys to the receiving side whether they approve of the information or not. The conveying methods include, as methods of stimulating people's five senses, for example, information transmission by voice, information transmission by light emission, stimulation transmission by vibration, etc. To show a more specific example, for example, in the case of a wearable device, since the measurement of biological information such as brain waves (measurement of the information content to be conveyed) and information transmission by voice can be simultaneously realized by utilizing the part of the ear, it is excellent in that BMI can be realized without preparing various devices. Here, the voice may be, in addition to language information, for example, music, chime, etc., as long as what it means is mutually defined in advance. There are also means of conveying the degree or extent of intention by, for example, notifying the intention of approval with two chime sounds or changing the sound range. For example, if it is a low-pitched chime sound, the degree of approval is low, but if it is a high-pitched chime sound, the degree of approval is high, and the conveying methods become diversified. With the emergence of such BMI, human-to-human communication can acquire various transmission methods that utilize biological information such as brain waves in addition to the conventional language-centered transmission through voice or GUI (Graphical User Interface).

[0306] Hereinafter, with reference to FIGS. 27 and 28, a specific example of the process according to Example 20 will be described. FIGS. 27 and 28 show flowcharts indicating the flow of the specific example. FIG. 27 shows the flow of the process when operating the device. FIG. 28 shows the flow of the process when people communicate with each other.

[0307] As shown in FIG. 27, first, an operation mode (OP mode) is selected (S01), and the operation intention information of the user is extracted and detected (S02). As a result, the device to be operated and the operation content are extracted and associated with each other. Also, a method for determining the accuracy of the operation intention is presented to the user. When the accuracy of the operation intention is high (S03, high accuracy), an operation on the device is immediately executed (S04). Thereafter, the correctness of the operation is verified (S05). For example, the verification is performed by the user. When the operation is correct (S06, Yes), the device is operated in the same manner next time (S07). For example, the operation content is set in the device, and the same operation is performed next time. When the operation is incorrect (S06, No), the process proceeds to step S11. Also, when the accuracy of the operation intention is low (S03, low accuracy), the operation intention is confirmed using another method (S08). Note that even when the accuracy of the operation intention is high, the operation intention may be confirmed using another method. For example, information indicating another method is displayed on the display or output as sound from the earphone of the wearable device. As a result, another method is presented to the user. When the confirmed operation intention is correct (S09, Yes), that is, when the confirmed operation intention matches the operation intention the user had in mind, an operation on the device is executed (S10). When the confirmed operation intention is incorrect (S09, No), the process proceeds to step S11. In step S11, the reason why the operation intention was incorrect is analyzed. For example, the analysis is performed by the user. Based on the analysis result, a next determination method (that is, an operation determination method) is selected (S12). Also, the determination criterion for the operation intention is changed (S13), and the changed content is reflected in the processing after step S03. For example, the threshold value registered in the management table is changed, and based on the changed threshold value, confirmation of the operation intention, determination of accuracy, etc. are performed. Note that each of the above processes may be selected by the user.

[0308] Also, when people communicate with each other, as shown in FIG. 28, a communication mode (hereinafter referred to as the "CM mode") is selected (S20). For example, either an automatic selection mode or a manual selection mode is set by the user, and the CM mode is selected according to the setting. Next, a communication target person (hereinafter referred to as the "CM target person") is selected (S21). For example, in a mode where the CM target person is automatically selected, participants in a meeting or the like are selected as the CM target person. In a mode where the CM target person is selected by the user's manual operation, for example, candidates for the CM target person are presented to the user by the voice or display of a wearable device, etc., and the user selects the CM target person. Next, information that may be conveyed to the CM target person is selected by the user (S22). For example, information that may be conveyed to the CM target person is selected from among a group of intention information, a group of status information, etc. Also, information that the user does not want to convey to the CM target person may be selected. For example, information indicating emotion is selected as information that the user does not want to convey to the CM target person. Next, biological information is measured or observed, and based on the biological information, intention information, status information, etc. are extracted (S23). Next, information to be conveyed to the CM target person is detected from the extracted intention information, status information, etc. (S24). For example, a detection test is executed in advance, and intention information, status information, etc. similar to the information detected by the test (for example, intention information, status information, etc. with a similarity equal to or greater than a threshold value) are detected from the extracted intention information, status information, etc. Note that the detection method may be presented to the user. Then, the detected intention information, status information, etc. are conveyed to the CM target person. For example, the transmission method may be presented to the user, and the user may select the transmission method. The transmission method is, for example, transmission by vibration (for example, transmission by voice), transmission by light, etc. Information may be conveyed to the CM target person by language, music, a light emission signal, etc. When the information is conveyed to the CM target person, the user who is the sender may be notified that the transmission has been completed. This notification may be omitted. Also, after the process of step S24 is executed, the processes after step S03 shown in FIG. 27 may be executed.

[0309] Alternatively, the process according to Example 20 may be executed using a wearable device. Since biological information is measured with the wearable device worn on a person's ear, each piece of information can be conveyed to the user by voice while the biological information can be measured. Also, since the wearable device is worn on the ear, it has the advantage of looking natural compared to an electroencephalograph or the like worn on the head to measure brain waves or the like. Also, the accuracy of the above-described operation mode and CM mode can be improved with BCI using the wearable device.

[0310] In each of the above embodiments, all or part of the processes executed by the processor 26 of the information processing apparatus 10 may be executed by the processor 34 of the terminal device 18, or may be executed by the biological information measurement apparatus 12. Also, the process may be executed by one or a plurality of devices selected from the information processing apparatus 10, the terminal device 18, and the biological information measurement apparatus 12.

[0311] <Wearable device> Hereinafter, with reference to FIGS. 29 to 37, a wearable device which is an example of the biological information measurement apparatus 12 according to the present embodiment will be described in detail. Hereinafter, as an example of the wearable device, an earphone worn on a person's ear will be described as an example, but the wearable device described here may be a wearable device other than an earphone.

[0312] For example, one or a plurality of electrodes are used as bioelectric potential detection electrodes (hereinafter referred to as "sensor electrodes") for detecting a potential including a bioelectric potential, another one or a plurality of electrodes are used as electrodes for grounding (hereinafter referred to as "ground electrodes"), and still another one or a plurality of electrodes are used as electrodes for detecting a potential for comparing with the potential detected by the sensor electrodes (hereinafter referred to as "reference electrodes").

[0313] Hereinafter, the potential detected by the sensor electrode shall be referred to as the "sensor potential", the potential detected by the reference electrode shall be referred to as the "reference potential", and the potential detected by the ground electrode shall be referred to as the "ground potential".

[0314] FIG. 29 is a perspective view showing the entirety of the wearable device 200. FIG. 30 is a view of the wearable device 200 as seen from above. FIG. 31 is a view of the left earphone as seen from the left. FIG. 32 is a perspective view showing the right earphone. FIG. 33 is a perspective view showing the left earphone. FIG. 34 is a view of the right earphone as seen from above. FIG. 35 is an exploded perspective view of the right earphone. FIG. 36 is a cross-sectional view of the right earphone. FIG. 37 is a perspective view showing the inside of the left earphone.

[0315] Here, for the sake of convenience of explanation, as shown in FIG. 29, the front-rear direction, the up-down direction, and the left-right direction are defined. The front direction is the direction in which a person's face faces, and the rear direction is the direction opposite to the front direction. The up direction is the direction in which a person's head top faces, and the down direction is the direction opposite to the up direction. The right direction is the direction on a person's right hand side, and the left direction is the direction on a person's left hand side. The front-rear direction, the up-down direction, and the left-right direction are perpendicular to each other.

[0316] Here, as an example, the wearable device 200 is assumed to measure biological information including electroencephalogram. Of course, the wearable device 200 may measure electroencephalogram and / or measure biological information other than electroencephalogram without measuring only electroencephalogram. For example, when the right earphone 216R and the left earphone 216L described later are worn on a person's ears, a potential representing biological information is detected. The potential signal indicating the detected potential usually includes a potential signal representing electroencephalogram. In addition to the potential signal representing electroencephalogram, the potential signal may include a potential signal indicating a potential generated due to the movement of a person's head, such as a potential signal indicating a movement caused by a facial expression, a movement of the neck, jaw, or eyeballs. Also, as biological information caused by a change in blood flow accompanying the movement of a person's head, cerebral blood flow, a pulse wave related to the cardiac blood vessels, a heartbeat, etc. may also be significantly measurable in some cases. Thus, the wearable device 200 may measure biological information other than electroencephalogram together with electroencephalogram. The biological information other than electroencephalogram may be processed as noise (for example, may be removed), or may be separated from electroencephalogram and used for some processing.

[0317] As shown in FIGS. 29 and 30, the wearable device 200 generally includes a right earphone 216R worn on a person's right ear, a left earphone 216L worn on a person's left ear, and a cable 218 that electrically connects the right earphone 216R and the left earphone 216L. The right earphone 216R and the left earphone 216L may transmit and receive signals and data via the cable 218. The cable 218 may be provided with a remote control for operating the wearable device 200.

[0318] Either one of the right earphone 216R or the left earphone 216L may function as a biological information measuring device for measuring biological information, or both the right earphone 216R and the left earphone 216L may function as biological information measuring devices.

[0319] For example, one earphone may have a sensor electrode, a reference electrode, and a ground electrode, and the other earphone may not have an electrode.

[0320] As another example, both the right earphone 216R and the left earphone 216L may have a sensor electrode, a reference electrode, and a ground electrode.

[0321] As yet another example, one earphone may have one or two electrodes out of the sensor electrode, the reference electrode, and the ground electrode, and the other earphone may have the electrodes that the one earphone does not have.

[0322] As yet another example, one or more electrodes are provided on each of the right earphone 216R and the left earphone 216L, and each of the sensor electrode, the reference electrode, and the ground electrode may be assigned to any of the electrodes.

[0323] As yet another example, a plurality of sensor electrodes, a plurality of reference electrodes, and a plurality of ground electrodes may be provided on one earphone. For example, a plurality of sensor electrodes may be provided on the right earphone 216R, and a plurality of reference electrodes may be provided on the left earphone 216L.

[0324] The right earphone 216R is, for example, a canal-type earphone and includes a right housing 220R, a right earpiece 222R, a right support 224R, a right earhook 226R, and an electrode member 228R. The earpiece may also be referred to as an ear pad.

[0325] The right housing 220R has, for example, a thin rectangular parallelepiped shape and is a housing for storing an electronic circuit or the like. In the right housing 220R, a right earpiece 222R and a right support 224R are provided on the surface facing a person's right ear when the user wears the wearable device 200. Inside the right housing 220R, a control device, a speaker, a communication device (for example, a communication chip), an electronic circuit for analyzing and processing biological information, a six-axis sensor (for example, a sensor having a three-axis sensor for detecting acceleration and a three-axis sensor for detecting angular velocity), a memory, and a GPS (Global Positioning System) sensor or the like are stored. The communication device is a device for realizing a wireless communication function such as Bluetooth (registered trademark) or BLE (Bluetooth (registered trademark) Low Energy). This communication device is equipped with modules for wireless LAN (for example, a network using WiFi or the like), cellular (3G, 4G, 5G, LPWA, etc.) to expand the communication area, and can directly transmit data to devices in a local area with a longer communication distance than Bluetooth (registered trademark) or to remote devices via the Internet. The movement direction, orientation, and rotation of the right housing 220R are detected by the six-axis sensor. The biological information may be stored in the memory. Note that the electronic circuit for analyzing the biological information may not be provided in the biological information measuring device 12.

[0326] The right support 224R has a columnar shape such as a cylinder and protrudes from the side surface of the right housing 220R (for example, the surface facing the right ear when the right earphone 216R is worn on the right ear), and is provided between the right housing 220R and the right earpiece 222R. For example, the outer diameter of the right support 224R is larger than the outer diameter of the right earpiece 222R. An electrode member 228R is provided on a part or all of the side surface of the right support 224R.

[0327] For example, the electrode member 228R has a ring shape and is supported by the columnar right support 224R. All or part of the electrode member 228R functions as an electrode. That is, electrodes may be provided on the entire surface of the electrode member 228R, or electrodes may be provided on a part of the surface of the electrode member 228R, and electrodes may not be provided on the surface of that part. The electrode member 228R is composed of, for example, conductive rubber made of carbon. Even in this case, all of the electrode member 228R may be composed of conductive rubber and all of the electrode member 228R (for example, the entire surface) may function as an electrode, or a part of the electrode member 228R may be composed of conductive rubber and a part of the electrode member 228R (for example, a part of the surface) may function as an electrode.

[0328] The right earpiece 222R is provided at the tip of the right support 224R (that is, the end opposite to the end connected to the right housing 220R). For example, the right earpiece 222R has a columnar shape that narrows in width toward the tip. Of course, this shape is just an example, and the right earpiece 222R may have another shape.

[0329] Inside the right earpiece 222R, a sound guide tube or the like (for example, a conductive tube member described later) is stored, and the sound emitted by the speaker is transmitted through the sound guide tube or the like inside the right earpiece 222R and emitted from the right earpiece 222R to the outside. Electrodes are provided on a part or all of the outer surface (for example, the side surface) of the right earpiece 222R. The electrode is composed of, for example, conductive rubber made of carbon. The right earpiece 222R itself may be composed of conductive rubber. For example, all or part of the right earpiece 222R may be composed of conductive rubber. That is, the entire surface or a part of the surface of the right earpiece 222R may function as an electrode.

[0330] The right earpiece 222R and the electrode member 228R may be constituted by an elastic member, for example. As the elastic member, a resin such as rubber is used, for example. Specifically, Si-based rubber (for example, S1734 manufactured by NOK Corporation) or urethane-based rubber may be used for the right earpiece 222R and the electrode member 228R. Further, the hardness of each of the right earpiece 222R and the electrode member 228R (for example, the hardness according to the standard of durometer type A (instantaneous)) is, for example, 40 to 75. As an example, a resin having a hardness of 70 may be used for the right earpiece 222R and the electrode member 228R.

[0331] As will be described later, the right earpiece 222R is arranged on the right ear so as to be inserted into the external auditory canal of the right ear and contact the external auditory canal, and the electrode member 228R supported by the right support 224R is arranged on the right ear so as to contact the concha cavity of the right ear.

[0332] The right ear hook 226R has a curved shape as a whole and is a member that is hung on the user's right ear when the user wears the right earphone 216R. For example, the right ear hook 226R is hung on the helix of the right ear. More specifically, the right ear hook 226R is provided in contact with the helix on the back side of the helix. One end of the right ear hook 226R is connected to the front part of the right housing 220R, and the right ear hook 226R has a curved shape from the connection part to the rear side of the right housing 220R, and that part constitutes a curved part. The curved part is provided in contact with the helix on the back side of the helix of the right ear. For example, the curved part has a shape along the shape on the back side of the helix. The other part of the right ear hook 226R is connected to one end of the cable 218.

[0333] The right earpiece 222R and the right support 224R are replaceable members. For example, a plurality of types (for example, 3 to 5 types, etc.) of right earpieces 222R and right supports 224R having different shapes and sizes are prepared in advance, and the right earpiece 222R and the right support 224R can be exchanged according to the shape of the user's ear (for example, the external auditory canal, the concha cavity, and other parts).

[0334] The left earphone 216L is, for example, a canal-type earphone and includes a left housing 220L, a left earpiece 222L, a left support 224L, a left ear hook 226L, and an electrode member 228L.

[0335] The left housing 220L has, for example, a thin rectangular parallelepiped shape and is a housing for storing a battery or the like. In the left housing 220L, the left earpiece 222L and the left support 224L are provided on the surface facing the user's left ear when the user wears the wearable device 200. Power from the battery is supplied to the right earphone 216R and the left earphone 216L, thereby driving the right earphone 216R and the left earphone 216L. For example, power from the battery is supplied to a speaker and each circuit. Note that the battery may be provided in both the right housing 220R and the left housing 220L, or may be provided in either the right housing 220R or the left housing 220L.

[0336] The left support 224L has a columnar shape such as a cylinder and protrudes from the side surface of the left housing 220L (for example, the surface facing the left ear when the left earphone 216L is worn on the left ear), and is provided between the left housing 220L and the left earpiece 222L. For example, the outer diameter of the left support 224L is larger than the outer diameter of the left earpiece 222L. The electrode member 228L is provided on a part or all of the side surface of the left support 224L.

[0337] For example, the electrode member 228L has a ring shape and is supported by the columnar left support 224L. The entire electrode member 228L or a part thereof functions as an electrode. That is, electrodes may be provided on the entire surface of the electrode member 228L, or electrodes may be provided on a part of the surface of the electrode member 228L, and electrodes may not be provided on the surface of that part. The electrode member 228L is composed of, for example, conductive rubber made of carbon. Even in this case, the entire electrode member 228L may be composed of conductive rubber and the entire electrode member 228L (for example, the entire surface) may function as an electrode, or a part of the electrode member 228L may be composed of conductive rubber and a part of the electrode member 228L (for example, a part of the surface) may function as an electrode.

[0338] The left earpiece 222L is provided at the tip of the left support 224L (that is, the end opposite to the end connected to the left housing 220L). For example, the left earpiece 222L has a columnar shape that narrows in width toward the tip. Of course, this shape is only an example, and the left earpiece 222L may have another shape.

[0339] A sound guide tube or the like is stored in the left earpiece 222L, and the sound emitted by the speaker is transmitted through the sound guide tube or the like in the left earpiece 222L and emitted from the left earpiece 222L to the outside. Electrodes are provided on a part or all of the outer surface (for example, the side surface, etc.) of the left earpiece 222L. The electrode is composed of, for example, conductive rubber made of carbon. The left earpiece 222L itself may be composed of conductive rubber. For example, the entire left earpiece 222L or a part thereof may be composed of conductive rubber. That is, the entire surface or a part of the surface of the left earpiece 222L may function as an electrode.

[0340] The left earpiece 222L and the electrode member 228L may be constituted by, for example, an elastic member. As the elastic member, for example, a resin such as rubber is used. Specifically, Si-based rubber (for example, S1734 manufactured by NOK Corporation) or urethane-based rubber may be used for the left earpiece 222L and the electrode member 228L. Further, the hardness of each of the left earpiece 222L and the electrode member 228L (for example, the hardness according to the standard of durometer type A (instantaneous)) is, for example, 40 to 75. As an example, a resin having a hardness of 70 may be used for the left earpiece 222L and the electrode member 228L.

[0341] As will be described later, the left earpiece 222L is arranged on the right ear so as to be inserted into the external auditory canal of the right ear and contact the external auditory canal, and the electrode member 228L supported by the left support 224L is arranged on the left ear so as to contact the concha cavity of the left ear.

[0342] The left earhook 226L has a curved shape as a whole and is a member that is hung on the left ear of the user when the user wears the left earphone 216L. For example, the left earhook 226L is hung on the helix of the left ear. More specifically, the left earhook 226L is provided in contact with the helix on the back side of the helix. One end of the left earhook 226L is connected to a front portion of the left housing 220L, and the left earhook 226L has a curved shape extending from the connection portion to the rear side of the left housing 220L, and that portion constitutes a curved portion. The curved portion is provided in contact with the helix on the back side of the helix of the left ear. For example, the curved portion has a shape along the shape on the back side of the helix. The other portion of the left earhook 226L is connected to one end of the cable 218.

[0343] The left earpiece 222L and the left support 224L are replaceable members. For example, a plurality of types (for example, 3 to 5 types, etc.) of left earpieces 222L and left supports 224L having different shapes and sizes are prepared in advance, and the left earpiece 222L and the left support 224L can be exchanged according to the shape of the user's ear (for example, the external auditory canal, the concha cavity, and other parts).

[0344] Note that the control device, communication device, electronic circuit, six-axis sensor, memory, etc. may be stored in either the right housing 220R or the left housing 220L, or may be stored in both the right housing 220R and the left housing 220L.

[0345] In addition, the right housing 220R or the left housing 220L is provided with a power button, a switch for adjusting the volume, etc. Both the right housing 220R and the left housing 220L may be provided with a power button, a switch, etc.

[0346] For example, either one of the electrodes provided on the right earpiece 222R or the electrodes provided on the left earpiece 222L is used as a sensor electrode, and the other electrode is used as a reference electrode. In addition, the electrode members 228R and 228L are used as ground electrodes. Of course, either one of the electrode member 228R or the electrode member 228L may be used as a sensor electrode, the other may be used as a reference electrode, and the electrodes provided on the right earpiece 222R and the left earpiece 222L may be used as ground electrodes.

[0347] As another example, a plurality of electrodes separated from each other are provided on the right earpiece 222R, and at least one of the plurality of electrodes may be used as a sensor electrode, a reference electrode, or a ground electrode. For example, when the plurality of electrodes are used as sensor electrodes, the potential detected by the electrode with the highest detection sensitivity, the least noise, or the most stable noise level among the plurality of electrodes may be used as the sensor potential. The same applies when the plurality of electrodes are used as reference electrodes or ground electrodes. The same also applies to the left earpiece 222L, the electrode member 228R, and the electrode member 228L.

[0348] As shown in FIG. 34, the electrode member 228R supported by the right support 224R is installed at a position of height h from the right housing 220R. That is, the distance between the electrode member 228R and the right housing 220R is set to the height h. The height h is set to a value that, for example, prevents interference between the right housing 220R and the auricle and prevents poor contact. Specifically, when the right earpiece 222R is inserted into the external auditory canal of the right ear and the right earphone 216R is worn on the right ear, the height h is set so that the right housing 220R does not contact the auricle of the right ear. That is, the height h is set so as to be separated from the auricle to a position where the right housing 220R does not contact the auricle. By doing so, for example, the upper surface of the right housing 220R does not contact the auricle. If the right housing 220R contacts the auricle and the right housing 220R and the auricle interfere with each other, the right earpiece 222R may not be inserted to a position where it is stable in the external auditory canal, and poor contact may occur between the electrode of the right earpiece 222R and the external auditory canal. In addition, the electrode member 228R may not be disposed at a position where it makes good contact with the concha cavity of the right ear, and poor contact may occur between the electrode member 228R and the concha cavity of the right ear. When such poor contact occurs, the detection sensitivity of the sensor potential, the reference potential, and the ground potential may decrease, and the measurement accuracy of the biological information may decrease. By setting the height h so that the right housing 220R does not contact the auricle, the right housing 220R and the auricle do not interfere with each other, the right earpiece 222R can be inserted to a position where it is stable in the external auditory canal, and the contact between the electrode of the right earpiece 222R and the external auditory canal can be improved. In addition, the electrode member 228R can be disposed at a position where it makes good contact with the concha cavity of the right ear, and the contact between the electrode member 228R and the concha cavity can be improved. By doing so, the detection sensitivity of the bioelectric potential can be improved. The same applies to the left earphone 216L.

[0349] Also, as shown in FIG. 30, when viewed from above, the right earhook 226R is displaced in the direction in which the right earpiece 222R is installed from the position of the right housing 220R. That is, when the right earphone 216R is viewed from above, a gap is formed between the right housing 220R and the right earhook 226R. It can be said that the right earhook 226R is offset and arranged in the direction in which the right earpiece 222R is installed from the position of the right housing 220R. That gap is an ear space, and the right ear is inserted and arranged in that gap. By providing such an ear space and inserting and arranging the right ear in that ear space, regardless of the size of the right ear, the right ear can be arranged in the ear space and the right earphone 216R can be stably worn on the right ear. As a result, the contact between the electrode of the right earpiece 222R and the external auditory canal, and the contact between the electrode member 228R and the concha cavity can be improved. The same applies to the left earphone 216L.

[0350] As shown in FIG. 35, a conductive tube member 230R made of metal is provided on the right support 224R so as to protrude in a direction away from the right housing 220R. The rear end of the conductive tube member 230R is installed on the right support 224R, and a right earpiece 222R is provided at the tip of the conductive tube member 230R. For example, the rear end of the conductive tube member 230R is attached to the tip of the right support 224R by screwing or the like. On the side surface of the right support 224R, a conductive tube member 232R made of metal is provided in a direction protruding from the side surface. Also, on the side surface, a ring-shaped electrode member 228R is provided so as to cover the conductive tube member 232R. More specifically, a groove is formed along the circumferential direction of the right support 224R on the side surface of the right support 224R, and a hole 224R1 into which one end of the conductive tube member 232R is inserted is formed in the groove. By inserting one end of the conductive tube member 232R into the hole 224R1, the conductive tube member 232R is attached to the side surface of the right support 224R. For example, the conductive tube member 232R is attached to the side surface of the right support 224R by screwing or the like. With the conductive tube member 232R attached to the side surface of the right support 224R, the electrode member 228R is fitted into the groove on the side surface of the right support 224R, whereby the electrode member 228R is attached to the side surface of the right support 224R in a state of being in contact with the conductive tube member 232R. The conductive tube member 232R is connected to an electric wire inside the right support 224R. Thereby, a potential signal indicating the potential detected by the electrode member 228R is sent to the electric wire via the conductive tube member 232R, and is output to a main board 234 described later via the electric wire. The left earphone 216L also has the same configuration as the right earphone 216R.

[0351] Also, as shown in FIG. 36, a main board 234 and a sub-board 236 are housed in the right housing 220R. Sub-boards other than the main board 234 and the sub-board 236 may be housed in the left housing 220L.

[0352] Also, as shown in FIG. 37, a battery 238 and a relay board 240 are housed inside the left housing 220L. Electric power is supplied from the battery 238 to the right earphone 216R and the left earphone 216L via the relay board 240.

[0353] The cable 218 has a hardness that can maintain the overall shape of the cable 218. For example, when the wearable device 200 is worn on a person's ear, the cable 218 extends rearward from the right housing 220R and the left housing 220L so that the cable 218 does not contact the person's occipital region or the hair on the occipital region. In this way, since the cable 218 does not contact the occipital region or the hair on the occipital region, noise caused by the contact can be prevented from occurring in the biological information.

[0354] Even if the cable 218 contacts the occipital region or the like, the load due to the contact is transmitted to the cable 218, and as indicated by the arrow Y in FIG. 31, the entire left earphone 216L rotates rearward. As a result, the left earpiece 222L and the electrode member 228L are more firmly fixed to the left ear. That is, the entire left earphone 216L rotates in a direction that more firmly fixes the left earpiece 222L and the electrode member 228L. As a result, the contact between the electrode of the left earpiece 222L and the external auditory canal, and the contact between the electrode member 228L and the concha cavity can be improved, and the detection sensitivity can be enhanced. The same applies to the right earphone 216R.

[0355] Note that the wearable device 200 may be a device capable of outputting bone conduction. Thereby, information can be transmitted to a person by bone conduction instead of sound. For example, even for a user who cannot hear, by using the wearable device 200, biological information such as brain waves can be measured and information can be transmitted to the user by bone conduction.

[0356] Each of the above-described embodiments may be executed using the wearable device 200. That is, the biological information of the user may be measured using the wearable device 200, and each embodiment may be executed using the measured biological information. Of course, by using the wearable device 200 and the other biological information measuring device 12, a plurality of types of biological information may be measured, and based on the measured plurality of types of biological information, the state of a person (for example, emotion information, mental information, psychological information, etc.) may be obtained.

[0357] In each of the above embodiments, the processor refers to a processor in a broad sense, and includes a general-purpose processor (for example, CPU: Central Processing Unit, etc.) and a dedicated processor (for example, GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.). Further, the operation of the processor in each of the above embodiments may be achieved not only by one processor, but also by a plurality of physically separated processors cooperating with each other. Also, the order of each operation of the processor is not limited to the order described in each of the above embodiments, and may be changed as appropriate.

Description of Reference Numerals

[0358] 10 Information processing device, 12 Biological information measuring device, 14 Environment information measuring device, 16 Device, 18 Terminal device.

Claims

1. The processor: Acquire biometric information of the user; determining a state of the user according to context information when a specific waveform is measured among the time points when the biological information is measured; Information processing methods.

2. The information processing method according to claim 1 , further comprising the step of outputting information by audio means in accordance with the determined state of the user.

3. The information processing method according to claim 1 , further comprising the step of: executing control for calling another user in response to the determined state of the user.

4. The information processing method according to claim 1 , further comprising changing settings of a function that utilizes AI according to the determined state of the user.

Images