Information Processing Method

Abstract

To provide an information processing method for displaying movement specified by measuring biological information and advice corresponding to an image to be an object of the movement on a screen.SOLUTION: An information processing method makes a computer execute processing including extracting biological information and specific movement of the head from a plurality of biopotential measurement results corresponding to time series information including at least myoelectric information obtained by measuring bioelectric potential in the head of a human body, displaying the extracted biological information and the specific movement of the head in association with an image related to the head of a user on the screen, and displaying advice information corresponding to the specific movement on the screen.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing method. [Background technology]

[0002] Patent Document 1 discloses that its objective is to provide an electroencephalogram data acquisition device capable of measuring high-frequency electroencephalograms during REM sleep, and that signals from electrodes and sensors for collecting electroencephalograms, electromyograms (EMG), and electrooculograms (EOG) are passed through an amplifier and converted into digital data in an A / D converter at a fixed sampling rate from a sampling control unit and stored in a waveform memory unit, with these operations being controlled by an information processing unit, and that the waveform data stored in the waveform memory unit is printed out as waveform data in a recording unit, for example, via an interface unit, and that the sampling control unit increases the sampling rate when the data from the electromyogram (EMG) becomes flat and REM sleep is detected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2001-008915 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an information processing method for displaying, on a screen, a movement identified by measuring biological information and advice corresponding to an image that is the subject of the movement. [Means for solving the problem]

[0005] The gist of the present invention for achieving the above object resides in the following inventions. The invention of claim 1 is a computer that extracts bioinformation and a specific head movement from a plurality of bioelectric potential measurement results corresponding to time-series information including at least myoelectric information obtained by measuring the bioelectric potential in the human head, associates the extracted bioinformation and the specific head movement with an image related to the user's head and displays the advice information corresponding to the specific movement on the screen. Then, the electromyographic information is analyzed to detect that a specific movement of the head has been made, and an effect estimation is performed to notify the user of the training effect. The present invention relates to an information processing method for performing a process including the steps of:

[0006] The invention of claim 2 is as follows: An information processing method that executes a process including: extracting bioinformation and a specific head movement from a plurality of biopotential measurement results corresponding to time-series information including at least myoelectric information obtained by measuring a biopotential in a human body's head; displaying the extracted bioinformation and the specific head movement on a screen in association with an image related to a user's head; displaying advice information corresponding to the specific movement on the screen; and displaying an image of the user before the user performs the specific movement and an image of the user after the user performs the specific movement so as to be comparable. It is. Effect of the Invention

[0011] According to the information processing method of claim 1, it is possible to notify the biometric information and head movement on the same screen in a comparable manner, compared to a case in which the biometric information and head movement are measured by separate sensors.

[0012] According to the information processing method of claim 2, Displaying an image of a user in a manner that allows comparison of changes before and after the user performs a particular movement can be done. [Brief description of the drawings]

[0017] [Figure 1] FIG. 2 is a conceptual module configuration diagram of a configuration example of the present embodiment. [Figure 2A] FIG. 1 is an explanatory diagram showing an example of a system configuration using the present embodiment. [Figure 2B] FIG. 1 is an explanatory diagram showing an example of a system configuration using the present embodiment. [Figure 2C] FIG. 1 is an explanatory diagram showing an example of a system configuration using the present embodiment. [Diagram 3] FIG. 1 is an explanatory diagram showing a specific example of a system configuration using the present embodiment. [Figure 4] 4 is a flowchart showing an example of processing according to the present embodiment. [Diagram 5] 4 is a flowchart showing an example of processing according to the present embodiment. [Figure 6] 4 is a flowchart showing an example of processing according to the present embodiment. [Figure 7] 4 is a flowchart showing an example of processing according to the present embodiment. [Figure 8] FIG. 2 is an explanatory diagram showing a processing example according to the present embodiment. [Figure 9] FIG. 2 is an explanatory diagram showing a processing example according to the present embodiment. [Figure 10] FIG. 2 is an explanatory diagram showing a processing example according to the present embodiment. [Figure 11] FIG. 2 is an explanatory diagram showing a processing example according to the present embodiment. [Figure 12] FIG. 2 is an explanatory diagram showing a processing example according to the present embodiment. [Figure 13] FIG. 2 is an explanatory diagram showing a processing example according to the present embodiment. [Figure 14] FIG. 2 is an explanatory diagram showing a processing example according to the present embodiment. [Figure 15] FIG. 2 is an explanatory diagram showing a processing example according to the present embodiment. [Figure 16] FIG. 2 is an explanatory diagram showing a processing example according to the present embodiment. [Figure 17] FIG. 2 is an explanatory diagram showing a processing example according to the present embodiment. [Figure 18] FIG. 2 is a block diagram showing an example of a hardware configuration of a computer for implementing the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, a preferred embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows a conceptual module configuration diagram of an example of the configuration of this embodiment. In addition, a module generally refers to a logically separable piece of software (including a computer program as an interpretation of "software"), hardware, etc. Therefore, the module in this embodiment refers not only to a module in a computer program, but also to a module in a hardware configuration. Therefore, this embodiment also describes a computer program for functioning as those modules (for example, a program for causing a computer to execute each procedure, a program for causing a computer to function as each means, a program for causing a computer to realize each function), a system, and a method. However, for convenience of explanation, words such as "store" and "store" are used, but these words mean to store in a storage device or to control to store in a storage device when the embodiment is a computer program. In addition, a module may correspond one-to-one to a function, but in implementation, one module may be configured with one program, multiple modules may be configured with one program, or conversely, one module may be configured with multiple programs. In addition, multiple modules may be executed by one computer, or one module may be executed by multiple computers by computers in a distributed or parallel environment. In addition, one module may contain other modules. In the following, the term "connection" is used not only for physical connection but also for logical connection (e.g., data transfer, instruction, reference relationship between data, login, etc.). "Predetermined" means that it is determined before the target processing, and is used to include the meaning that it is determined according to the situation or state at that time or according to the situation or state up to that point, not only before the processing according to this embodiment starts, but also after the processing according to this embodiment starts, so long as it is before the target processing. When there are multiple "predetermined values", each may be a different value, or two or more values ​​("two or more values" includes all values, of course) may be the same. In addition, the description "If A, do B" is used to mean "determine whether A is true, and if it is determined that A is true, do B."However, this does not include cases where it is not necessary to judge whether it is A or not. In addition, when things are listed as "A, B, C", etc., this is an illustrative list unless otherwise specified, and includes cases where only one of them is selected (for example, only A). Furthermore, a system or device may be configured by connecting multiple computers, hardware, devices, etc. via a communication means such as a network ("network" includes one-to-one communication connections), and may also be realized by a single computer, hardware, device, etc. "Device" and "system" are used as synonymous terms. Needless to say, a "system" does not include anything that is merely a social "mechanism" (i.e., a social system) that is an artificial arrangement. Furthermore, for each process by each module, or for each process when multiple processes are performed within a module, the target information is read from the storage device, and after the process is performed, the processing results are written to the storage device. Therefore, the description of reading from the storage device before processing and writing to the storage device after processing may be omitted. Note that the storage device here may include a hard disk drive, RAM (short for Random Access Memory), an external storage medium, a storage device via a communication line, a register in the CPU (short for Central Processing Unit), etc.

[0019] The information processing device 100 according to this embodiment has a function of linking and notifying biometric information and head movement on the same screen, and as shown in the example of Fig. 1, has a biometric information extraction module 105, a head information extraction module 110, an analysis module 115, a display control module 120, and a display module 125. Note that "linking" means associating. Here, "biometric information" refers to information obtained by measuring the vital activities of the human body, and examples of the biometric information include electrocardiogram, heart rate, blood pressure, body temperature, electroencephalogram, electromyogram, retinal (fundus) potential, etc. In this embodiment, electroencephalogram information is mainly shown as an example. The term "head" refers to the part above the neck, including one or more of the following: ears, mouth, throat, eyes, nose, forehead, cheeks, etc. "Head movement" refers to the movement of the whole head or a part of the head. Examples include nodding, shaking the head, chewing, swallowing, blinking, breathing, and movement of the corners of the mouth. Specifically, the movement of the corners of the mouth may be determined from electromyographic information and a facial image. For this purpose, the face may be photographed at the same time as the measurement of the biometric information, and these may be linked to display and notify the display of the electric potential data and the face.

[0020] The biological information extraction module 105 is connected to the analysis module 115. The biological information extraction module 105 extracts biological information from a potential measurement result that is a result of measuring a potential at the head of a human body. Furthermore, the biological information extraction module 105 may extract waves in a plurality of frequency bands from the potential measurement results. Here, "waves in a frequency band" include, for example, α waves (alpha waves), β waves (beta waves), γ waves (gamma waves), θ waves (theta waves), and δ waves (delta waves), and "waves in multiple frequency bands" refers to two or more of these waves. Furthermore, as a process of the biological information extraction module 105, an electroencephalogram may be extracted by performing FFT (short for Fast Fourier Transform) on the potential measurement results.

[0021] The head information extraction module 110 is connected to the analysis module 115. The head information extraction module 110 extracts the movement of the head. The head information extraction module 110 may also extract head movement from potential measurement results that are the results of measuring the potential at the head of the human body.

[0022] The analysis module 115 is connected to the biometric information extraction module 105, the head information extraction module 110, and the display control module 120. The analysis module 115 uses the electroencephalogram information, which is the biometric information extracted by the biometric information extraction module 105, and the head movement extracted by the head information extraction module 110, to analyze the relationship between the electroencephalogram information and the head movement. The analysis module 115 may also analyze user behavior.

[0023] The display control module 120 is connected to the analysis module 115 and the display module 125. The display control module 120 controls the biometric information extracted by the biometric information extraction module 105 and the head movement extracted by the head information extraction module 110 so that they can be linked and notified on the same screen. Furthermore, the display control module 120 may perform control so that the multiple waves extracted by the biometric information extraction module 105 and the head movement are displayed on the screen of the display module 125 in a comparable manner.

[0024] The display module 125 is connected to the display control module 120. The display module 125 performs display on a screen such as a liquid crystal display or an organic EL display according to the control of the display control module 120.

[0025] For example, head movements may include chewing. In this case, the biological information extraction module 105 may extract, as the chewing part, a part where the potential measurement result matches a predetermined pattern. The "predetermined pattern" here may be such that the first peak of the graph is higher than or equal to a predetermined first threshold, the second peak after the first peak is lower than or equal to the first threshold, and is higher than or equal to a predetermined second threshold, which is lower than the first threshold. The display control module 120 may perform control so that the chewing location is displayed in a comparative manner on a graph showing the biological information and the potential measurement results.

[0026] Also, for example, head movement may include swallowing. In this case, the analysis module 115 may analyze a location where the potential measurement results match a predetermined pattern as being a swallowing location. Here, the "predetermined pattern" may be a state after chewing in which the third peak of the graph is higher than or equal to a predetermined third threshold value. The display control module 120 may perform control so that the swallowing location is displayed in a comparable manner on a graph showing the biological information and the potential measurement results. In this case, the analysis module 115 may further use the measurement results of an acceleration sensor attached to the head, and if the head is moving in a predetermined manner, extract the swallowing at a point where the potential measurement results match a predetermined pattern as the swallowing of a drink. The "predetermined movement" here may be a movement of tilting the head backward. Then, the display control module 120 may perform control to display the fact that the drink has been swallowed.

[0027] The display control module 120 may also display the results of multiple types of conversion processing of the potential measurement results on the same screen. For example, the display control module 120 may display the results of multiple types of processing of the acquired potential measurement results, such as FFT, wavelet transformation, Stockwell transformation, empirical mode decomposition (a function arbitrarily set by the user), on the same screen. Furthermore, the display control module 120 may display the intensity ratios as a result of frequency analysis as a graph as they are, and may also display the overall intensity ratios as a result of frequency analysis.

[0028] Furthermore, the display control module 120 may display the results of spectrum analysis of a predetermined period using a low-pass filter and a high-pass filter, as well as other data that does not have periodicity. Here, the "low-pass filter" is generally known as an LPF (short for Low-Pass Filter), and the "high-pass filter" is generally known as an HPF (short for High-Pass Filter).

[0029] The display control module 120 may also display graphs of the results of conversion processing using FFT or the like in an overlapping manner, allow the user to select the graph, and highlight the graph selected by the user. Examples of highlighting include changing the color or shape of the graph, or highlighting it.

[0030] In addition, the display control module 120 may be configured to allow the user to select a mode in which the vertical axis scale of the graph is adjusted according to the maximum and / or minimum value, a mode in which the display range of values ​​is arbitrarily fixed according to user operation, or a mode in which the scale is actively fixed at the maximum and / or minimum value.

[0031] 2A is an explanatory diagram showing an example of a system configuration using this embodiment. The system configuration shows an example of a system in which an information processing device 200A and a device 250 communicate with each other. Note that the information processing device 200A has the configuration of the information processing device 100. The same reference numerals are given to parts similar to those shown in the previous drawings, and duplicated explanations will be omitted (the same applies below).

[0032] The device 250 has a communication module 255 and a biological information detection module 260. The communication module 255 is connected to the biological information detection module 260, and is also connected to the communication module 230 of the information processing device 200A via a communication line. The communication module 255 communicates with the information processing device 200A. The communication line may be wireless, wired, or a combination of these. For example, the wireless line may be short-distance wireless communication such as Wi-Fi or Bluetooth (registered trademark). The biological information detection module 260 is connected to the communication module 255. The device 250 is attached to, for example, the user's head. The device 250 measures the electric potential at the head of the user who is wearing the device 250. For example, an electrode described in JP 2019-024758 A (an electrode made of a foam material, at least a portion that contacts the living body is conductive, and contacts the living body to detect brain waves) may be used. Then, the biological information detection module 260 passes the measurement result, which is the potential measurement result, to the communication module 255 and transmits it to the information processing device 200A.

[0033] The information processing device 200A includes a biometric information extraction module 105, a head information extraction module 110, an analysis module 115, a display control module 120, a display module 125, and a communication module 230.

[0034] The communication module 230 is connected to the bioinformation extraction module 105 and the head information extraction module 110, and is also connected to the communication module 255 of the device 250 via a communication line. The communication module 230 communicates with the device 250 and receives the potential measurement result. The potential measurement result received from the device 250 is passed to the bioinformation extraction module 105 and the head information extraction module 110.

[0035] The biological information extraction module 105 is connected to the analysis module 115 and the communication module 230. The biological information extraction module 105 extracts biological information from the potential measurement result received from the device 250, which is the result of measuring the potential at the head of the human body. The head information extraction module 110 is connected to the analysis module 115 and the communication module 230. The head information extraction module 110 extracts head movement from the potential measurement result received from the device 250, which is the result of measuring the potential at the head of the human body.

[0036] 2B is an explanatory diagram showing an example of a system configuration using this embodiment. The system configuration shows an example of a system in which an information processing device 200B communicates with a device 250. Note that the information processing device 200B has the configuration of the information processing device 100.

[0037] The information processing device 200B includes a biometric information extraction module 105, a head information extraction module 110, an analysis module 115, a display control module 120, a display module 125, and a communication module 230. The communication module 230 is connected to the biological information extraction module 105, and is also connected to the communication module 255 of the device 250 via a communication line. The communication module 230 communicates with the device 250 to receive the potential measurement result. The potential measurement result received from the device 250 is passed to the biological information extraction module 105.

[0038] The biological information extraction module 105 is connected to the analysis module 115 and the communication module 230. The biological information extraction module 105 extracts biological information from the potential measurement result received from the device 250, which is the result of measuring the potential at the head of the human body. The head information extraction module 110 has a photographing module 235 and is connected to the analysis module 115. The photographing module 235 photographs the head of a user who possesses the information processing device 200B (this user is also the user who wears the device 250). The photographed image may be a still image or a video. In the case of a still image, it is preferable to photograph two or more still images at different times. The head information extraction module 110 extracts head movement from an image of the user's head captured by the imaging module 235. For example, the head movement may be extracted from the image using a learning model generated by machine learning. The display control module 120 further controls to display the image captured by the image capture module 235. At this time, a graph showing the biometric information and the movement of the head may be displayed together with the image captured by the image capture module 235. Also, an image when there is a movement of the head may be displayed in conjunction with the image.

[0039] 2C is an explanatory diagram showing an example of a system configuration using this embodiment. The system configuration shows an example of a system in which an information processing device 200C communicates with a device 250. Note that the information processing device 200C has the configuration of the information processing device 100.

[0040] The information processing device 200C includes a biometric information extraction module 105, a head information extraction module 110, an analysis module 115, a display control module 120, a display module 125, and a communication module 230. The communication module 230 is connected to the bioinformation extraction module 105 and the head information extraction module 110, and is also connected to the communication module 255 of the device 250 via a communication line. The communication module 230 communicates with the device 250 and receives the potential measurement result. The potential measurement result received from the device 250 is passed to the bioinformation extraction module 105 and the head information extraction module 110.

[0041] The biological information extraction module 105 is connected to the analysis module 115 and the communication module 230. The biological information extraction module 105 extracts biological information from the potential measurement result received from the device 250, which is the result of measuring the potential at the head of the human body. The head information extraction module 110 has a photographing module 235 and is connected to the analysis module 115 and the communication module 230. The head information extraction module 110 extracts head movement from the potential measurement result received from the device 250 and the image of the user's head photographed by the photographing module 235. It has the functions of both the head information extraction module 110 shown in the example of FIG. 2A and the head information extraction module 110 shown in the example of FIG. 2B. For example, if the head movement extracted from the potential measurement result received from the device 250 and the head movement extracted from the image of the head match, it may be extracted as the head movement. If they differ, it may not be possible to extract it as the head movement, or the head movement resulting from either one of them may be adopted.

[0042] FIG. 3 is an explanatory diagram showing a specific example of a system configuration using this embodiment. The smartphone 300 is a specific example of the information processing device 200 (particularly, the information processing device 200B or the information processing device 200C), and the wearable device 350 is a specific example of the device 250. The smartphone 300 has a camera 335 and captures an image of the head of the user 390. The camera 335 is a specific example of the imaging module 235. A user 390 carries a smartphone 300 and wears a wearable device 350 on his / her head. The smartphone 300 and the wearable device 350 communicate with each other using short-range wireless communication. The smartphone 300 and the wearable device 350 correlate and display the user 390's own biometric information and head movement on the screen of the smartphone 300. For example, the chewing and swallowing of the user 390 are sensed. Wearable device 350 is a device in which a sensor is placed in the ear canal, and wearable device 350 detects data of a composite waveform of electroencephalograms and electromyograms (so-called raw data, hereinafter also referred to as raw data). The raw data is transmitted to smartphone 300. Smartphone 300 converts the raw data itself into electromyogram data, performs FFT on the raw data, and converts it into electroencephalogram data. In other words, two pieces of information are obtained from one waveform data (raw data), that is, electroencephalogram data, which is biological information, and electromyogram data, which indicates head movement. For example, smartphone 300 detects chewing and swallowing from the electromyogram data, marks the graph of the electromyogram data, and also displays the graph of the electroencephalogram data in the same time series as the graph of the electromyogram data, so that user 390 can know the state of the electroencephalogram data when chewing and swallowing were performed. Specifically, the system extracts peaks from the EMG data graph and compares them with a threshold to detect chewing and swallowing. Then, it marks chewing and swallowing on the EMG data graph. It also performs FFT analysis on the raw data to eliminate noise frequency components, analyzes electroencephalograms to generate an EEG graph, and displays it in a time series aligned with the EMG data graph.

[0043] In addition, the head, mainly the face, of the user 390 may be photographed using the camera 335 of the smartphone 300, and the relationship between the facial movement and the brain waves may be correlated and displayed. Examples of the facial movement include beauty facial exercises and training of facial muscles.

[0044] When the electrode described in the aforementioned JP 2019-024758 A is used as a sensor of the wearable device 350, the sensor is an electrometer that measures minute electric potentials (e.g., at the level of a few μV), and it is required to obtain a signal with a high S / N ratio even with slight changes in electric potential. However, because it can detect even the slightest changes in potential, myoelectric signals generated by the muscles around the electrodes are also picked up as noise in the signal. In this embodiment, myoelectric information, which is usually treated as noise, is used to detect head movements, such as chewing and swallowing. Wearable device 350 measures brainwaves by placing electrodes in the ear canal and a ground (GND) in the area around the ear. The large muscles of the jaw, cheeks, and throat are located nearby, making it possible to obtain electromyographic signals from actions such as chewing and swallowing. When the electrodes described in JP 2019-024758 A are used, the following process is performed in this embodiment. (1) The EEG electrodes of the wearable device 350 are placed in the ear canal of the user 390. (2) Raw data is converted into electromyogram data, and the raw data is subjected to FFT processing to obtain EEG data, thereby obtaining two pieces of biometric information from one waveform data.

[0045] FIG. 4 is a flowchart showing an example of processing according to this embodiment (wearable device 350). In step S402, the wearable device 350 detects biometric information of the user 390. This becomes the aforementioned raw data. In step S404, data to be transmitted to the smartphone 300 is generated. In step S406, the data is transmitted to the smartphone 300. As described above, for example, the smartphone 300 and the wearable device 350 perform short-range wireless communication.

[0046] FIG. 5 is a flowchart showing an example of processing according to this embodiment (smartphone 300). In step S502, data is received from the wearable device 350. In step S504, information relating to brain waves is extracted from the received data. In step S506, information regarding head movement is extracted from the received data. In step S508, graphs of both the information on brain waves and the information on head movement are displayed on the same screen. Of course, the time series of the two graphs are aligned. Specifically, the two graphs may be displayed with the X-axis indicating time aligned, or the two graphs may be displayed overlapping each other in one area.

[0047] FIG. 6 is a flowchart showing an example of processing according to this embodiment (smartphone 300). In step S602, data is received from the wearable device 350. In step S604, an image captured by the camera 335 of the smartphone 300 is acquired. In step S606, information relating to brain waves is extracted from the received data. In step S608, the image is analyzed to extract information regarding head movement. In step S610, graphs of both the information related to the brain waves and the information related to the head movement are displayed on the same screen. For example, the time points at which head movements occur may be marked on the graph of the brain waves.

[0048] 7 is a flowchart showing an example of processing according to this embodiment (smartphone 300), which is a combination of the example of processing shown in the example of FIG. 5 and the example of processing shown in the example of FIG. In step S702, data is received from the wearable device 350. In step S704, an image captured by the camera 335 is acquired. In step S706, information relating to brain waves is extracted from the received data. In step S708, information regarding head movement is extracted from the received data. In step S710, the images are analyzed to extract information regarding head movement. In step S712, the information relating to head movement extracted in steps S708 and S710 is combined. In step S714, graphs of both the information relating to electroencephalograms and the information relating to head movement are displayed on the same screen.

[0049] FIG. 8 is an explanatory diagram showing a processing example according to this embodiment. For example, a raw data field 802, an FFT field 804, a wavelet transform field 806, a Stockwell transform field 808, and an empirical mode decomposition field 810 are displayed on a screen 800 of the smartphone 300. Signals acquired from wearable device 350 are subjected to multiple types of processing, such as FFT, wavelet transform, Stockwell transform, and empirical mode decomposition (a function arbitrarily set by the user), and the results are displayed on screen 800. Here, the horizontal axis, which is the time axis, is the same, and each graph is displayed separately in a raw data column 802, an FFT column 804, a wavelet transform column 806, a Stockwell transform column 808, and an empirical mode decomposition column 810. Also, the intensity ratios as a result of frequency analysis may be displayed as a graph, and the overall intensity ratios as a result of frequency analysis may be displayed.

[0050] 9 is an explanatory diagram showing a processing example according to the present embodiment. In this example, raw data of biometric information, analyzed data of the biometric information, and processed data of the biometric information (data obtained by removing periodic components from the raw data) are displayed. The screen 900 displays a raw data column 902, a frequency analysis column 904, a periodic frequency waveform generation column 906, and a processed waveform data column 908. Periodic frequency components appear in the graph in the frequency analysis field 904. In Fig. 9, the peak portion in the graph in the frequency analysis field 904 corresponds to this. The processed waveform data field 908 displays, for example, a graph generated from the graph in the raw data field 902 and the graph in the periodic frequency waveform generation field 906. In other words, by removing the periodic frequency components from the raw data, it becomes easier to see non-periodic waveforms such as myoelectric waves.

[0051] FIG. 10 is an explanatory diagram showing a processing example according to this embodiment. This shows an example of a graph with chewing and swallowing locations marked. A screen 1000 displays a raw data column 1002, a delta wave column 1004, a theta wave column 1006, an alpha wave column 1008, and an electromyographic data column 1010. The graphs in the delta wave column 1004, theta wave column 1006, and the alpha wave column 1008 are obtained by subjecting raw data to processing such as FFT and breaking it down into delta waves, theta waves, and alpha waves. The electromyographic data column 1010 displays a graph of electromyographic data extracted by removing electroencephalographic components from the raw data. The locations where chewing and swallowing occurred are marked on this graph. Regarding chewing, the first peak of the graph is higher than or equal to a predetermined first threshold, and the second peak following the first peak is lower than or equal to the first threshold, and is higher than or equal to a predetermined second threshold, where chewing has occurred. The second threshold is lower than the first threshold. In the example of FIG. 10, the first peak marked with a chewing mark is a peak higher than the first threshold, and the subsequent second peak satisfies the relationship of being lower than the first threshold and higher than the second threshold. Regarding swallowing, the location where swallowing has occurred is the location after chewing where the third peak is higher than or equal to the third threshold value. In the example of Fig. 10, the location of the third peak marked with the swallowing mark is after the first peak and satisfies the relationship of being higher than the third threshold value. Alternatively, the location where swallowing has occurred may be the location after the first peak, after the second peak, and satisfies the relationship of being higher than the third threshold value. By displaying the data in this way, when chewing and swallowing are detected from electromyography data, the relationship with each electroencephalogram can be confirmed on a single screen, making it easier to formulate hypotheses and verify them.

[0052] FIG. 11 is an explanatory diagram showing a processing example according to this embodiment. The graph shown in the example of Fig. 11(a) is a graph of raw data, that is, electroencephalogram data and electromyogram data are synthesized. The raw data is separated into electroencephalogram data and electromyogram data. For example, the raw data may be subjected to FFT processing to generate electroencephalogram data. The raw data itself may be treated as electromyogram data. In other words, two pieces of information are obtained from one piece of raw data obtained from the wearable device 350. 11(b) is a graph of myoelectric data. Here, chewing points 1150 and 1152 are higher than threshold 1110, and the peak immediately following them is higher than threshold 1120, so they are determined to be chewing points. The graph shown in the example of FIG. 11(c) is a graph of electroencephalogram data obtained by performing FFT analysis on the raw data and removing noise frequency components.

[0053] 12 is an explanatory diagram showing a processing example according to the present embodiment. Wearable device 350 may have not only a power-saving sensor but also an acceleration sensor. In that case, information may be displayed using the detection result by the acceleration sensor. The data of the acceleration data graph 1210 is acquired by an acceleration sensor in the wearable device 350 attached to the head of the user 390. For example, a six-axis sensor may be used as this acceleration sensor. The data in the raw data graph 1220 is obtained by a power saving sensor within the wearable device 350. The movement can be classified into more detailed movements by correlating the movement of the head with the electromyographic information of the jaw using the detection results of the acceleration sensor in the wearable device 350. For example, if information is obtained that the head has been tilted back during swallowing, it is determined that this is the movement of drinking a drink. In the example of Fig. 12, there are peaks in the acceleration data graph 1210 immediately before the chewing points 1250 and 1252 in the raw data graph 1220, which indicates that the head was tilted backward immediately before swallowing. Note that this graph shows acceleration in the front-back direction. Also, by using acceleration in the lateral direction, it is possible to determine whether the face was tilted, etc.

[0054] FIG. 13 is an explanatory diagram showing a processing example according to this embodiment. A raw data graph 1310, an electroencephalogram (α waves) graph 1320, and an electroencephalogram (β waves) graph 1330 are displayed in an overlapping manner on a screen 1300. Naturally, the horizontal axis, which is the time axis, is aligned. In this example, the user can select a graph to highlight that graph. For example, when the brain wave (alpha wave) graph 1320 is selected by the user, the brain wave (alpha wave) graph 1320 may be displayed in red. Then, by marking the places where chewing took place, such as chewing place 1350 and chewing place 1352, it becomes possible to observe the state of the electroencephalogram when chewing is taking place.

[0055] FIG. 14 is an explanatory diagram showing a processing example according to this embodiment. Smartphone 300 has a screen 325 and a camera 335. Camera 335 is also called an in-camera, and is used to capture an image of the face of user 390 who owns smartphone 300. An example will be shown in which the face of user 390 is captured by camera 335 and information detected by wearable device 350 is displayed in a graph. 14 shows an example of use for beauty face exercises, facial muscle training, etc. An example is shown in which the face image of a user 390 and the measurement results of biological information are displayed in conjunction with each other.

[0056] On the screen 325, a captured image display area 1410, a remaining number of times display area 1420, a remaining time display area 1430, a comment display area 1440, and a graph display area 1450 are displayed. When the wearable device 350 is initially set up, it is initialized according to the potential of each individual. When initializing, either or both of the potential and image data are used. When the potential cannot be measured, the number of training sessions is counted from image information only. For example, training may be performed using the orbicularis oculi muscle. Note that the case where the potential cannot be measured is an example of a case where biological information cannot be extracted. Screen 325 of smartphone 300 is displayed by a training application, and displays the face of the user 390, the electric potential data, and activity information in association with each other. Activity information includes, for example, advice such as opening your eyes more, the number of training sessions, and the training time. For example, remaining number of sessions display area 1420 displays "remaining number of sessions: 5 sessions," remaining time display area 1430 displays "remaining time: 00:32," and comment display area 1440 displays "please lift your cheeks more." Also, the electromyogram data from the wearable device 350 is analyzed to detect that lifting up or the like has been performed. Furthermore, a face image may be analyzed to detect that lifting up or the like has been performed. That is, even in an abnormal situation where information cannot be obtained from bioelectric potential such as electromyogram, the effect may be estimated without stopping the application, and the user 390 may be notified of the training effect. Similarly, when a face image cannot be obtained, the effect result may be estimated only from the information from the bioelectric potential, and notified to the user 390. Note that when both are normally measured and analyzed, the training effect can be estimated and notified with a higher accuracy than when only one of them is measured.

[0057] FIG. 15 is an explanatory diagram showing a processing example according to this embodiment. The screen 325a shown in Fig. 15(a) displays the face image before training, and the screen 325b shown in Fig. 15(b) displays the face image after training. In this example, it can be seen that the cheek line 1412a and the cheek line 1414a have been raised and changed to the cheek line 1412b and the cheek line 1414b by the training. In addition, the image comparison may be made selectable from, for example, displaying side-by-side on one screen, displaying one image at a time, combining two images with transparency, displaying half each of the left and right images before and after training, or partially enlarged. Also, the facial image captured when the face is lifted up may be displayed.Facial images taken before and after the lifting may also be displayed.

[0058] Fig. 16 is an explanatory diagram showing a processing example according to this embodiment. The example shown in Fig. 16 shows an example used for beauty face exercises, training of facial muscles, etc. An example is shown in which the face image of a user 390 and the measurement results of biological information are displayed in conjunction with each other. On the screen 325, a captured image display area 1610, a remaining number of times display area 1620, a remaining time display area 1630, a comment display area 1640, and a graph display area 1650 are displayed. Similar to the example shown in FIG. 14, the example shown in FIG. 16 relates to training the orbicularis oculi muscle. For example, the remaining number of times display area 1620 displays "Remaining number of times: 5 times", the remaining time display area 1630 displays "Remaining time: 00:32", and the comment display area 1640 displays "Please keep your eyes open longer."

[0059] FIG. 17 is an explanatory diagram showing a processing example according to this embodiment. The screen 325a shown in Fig. 17(a) displays a face image before training, and displays a captured image display area 1710a and a training target display area 1720a on the screen 325a. The screen 325b shown in Fig. 17(b) displays a face image after training, and displays a captured image display area 1710b and a training target display area 1720b on the screen 325b. In this example, it can be seen that the training significantly changes the size of the eyes. In particular, the training target area is enlarged. In this example, the image of the eye in the training target display area 1720a can be compared with the image of the eye in the training target display area 1720b. In addition, the image comparison may be made selectable from, for example, displaying side-by-side on one screen, displaying one image at a time, combining two images with transparency, displaying half each of the left and right images before and after training, or partially enlarged. Also, a facial image captured when the eyes are opened wide, etc. may be displayed. Furthermore, facial images before and after the opening may also be displayed.

[0060] The hardware configuration of the computer on which the program according to the present embodiment is executed is a general computer, specifically a personal computer, a computer that can be a server, etc., as exemplified in Fig. 18. That is, as a specific example, a CPU 1801 is used as a processing unit (arithmetic unit), and a RAM 1802, a ROM 1803, and a HDD 1804 are used as storage devices. For example, an HDD (short for Hard Disk Drive) or an SSD (short for Solid State Drive), which is a flash memory, etc. may be used as the HDD 1804. The computer is configured with a CPU 1801 for executing programs such as the bioinformation extraction module 105, the head information extraction module 110, the analysis module 115, the display control module 120, the display module 125, the communication module 230, the communication module 255, and the bioinformation detection module 260, a RAM 1802 for storing the programs and data, a ROM 1803 in which programs for starting the computer are stored, a HDD 1804 which is an auxiliary storage device for storing brainwave information, head information, and the like, a reception device 1806 for receiving data based on user operations (including motion, voice, line of sight, and the like) on a keyboard, a mouse, a touch screen, a microphone, a camera (including a line of sight detection camera, and the like), an output device 1805 such as a CRT, a liquid crystal display, a speaker, and the like, a communication line interface 1807 for connecting to a communication network such as a network interface card, and a bus 1808 for connecting them to exchange data. A plurality of these computers may be connected to each other via a network.

[0061] Of the above-mentioned embodiments, those that are implemented using computer programs are realized by loading the computer program, which is software, into a system having this hardware configuration, and the software and hardware resources work together to realize the above-mentioned embodiments. The hardware configuration shown in Fig. 18 is one example configuration, and the present embodiment is not limited to the configuration shown in Fig. 18, and may be configured in any configuration capable of executing the modules described in the present embodiment. For example, some of the modules may be configured with dedicated hardware (for example, an application specific integrated circuit (ASIC) or a reconfigurable integrated circuit (field-programmable gate array (FPGA)), etc.), some of the modules may be in an external system and connected via a communication line, and further, a plurality of the systems shown in Fig. 18 may be connected to each other via a communication line and operate in cooperation with each other. In particular, the systems may be incorporated in portable information and communication devices (including mobile phones, smartphones, mobile devices, wearable computers, etc.), information appliances, robots, etc.

[0062] The program described above may be provided by storing it in a recording medium, or the program may be provided via a communication means. In that case, for example, the program described above may be regarded as an invention of "a computer-readable recording medium on which a program is recorded." "Computer-readable recording medium on which a program is recorded" refers to a computer-readable recording medium on which a program is recorded, which is used for installing, executing, distributing, etc. a program. Examples of recording media include digital versatile discs (DVDs) such as "DVD-R, DVD-RW, DVD-RAM, etc." which are standards established by the DVD Forum, and "DVD+R, DVD+RW, etc." which are standards established for DVD+RW, compact discs (CDs) such as read-only memories (CD-ROMs), CD recordables (CD-Rs), CD rewritables (CD-RWs), Blu-ray (registered trademark) Discs, magneto-optical discs (MOs), flexible disks (FDs), magnetic tapes, hard disks, read-only memories (ROMs), electrically erasable and rewritable read-only memories (EEPROMs, registered trademarks), flash memories, random access memories (RAMs), and SD (short for Secure Digital) memory cards. The whole or a part of the program may be recorded on the recording medium and stored or distributed, etc. Also, the program may be transmitted by communication using a transmission medium such as a wired network used in a local area network (LAN), metropolitan area network (MAN), wide area network (WAN), the Internet, an intranet, an extranet, etc., a wireless communication network, or a combination of these, or may be transmitted on a carrier wave. Furthermore, the program may be a part or the whole of another program, or may be recorded on a recording medium together with a separate program, or may be divided and recorded on a plurality of recording media, or may be recorded in any manner that allows restoration, such as compression or encryption. [Explanation of symbols]

[0063] 100...Information processing device 105...Biometric information extraction module 110...Head information extraction module 115…Analysis module 120...Display control module 125...Display module 200...Information processing device 230…Communication module 235…Camera module 250...device 255…Communication module 260...Biometric information detection module 300…Smartphone 325…screen 335...Camera 350…Wearable devices 390…users

Claims

1. The computer extracting bioinformation and a specific movement of the head from a plurality of bioelectric potential measurement results corresponding to time-series information including at least myoelectric information obtained by measuring a bioelectric potential in the head of the human body; displaying the extracted biometric information and the specific head movement on a screen in association with an image relating to the user's head; displaying advice information corresponding to the specific movement on the screen; Analyzing the myoelectric information, detecting that a specific movement of the head has been made, estimating the effect, and notifying the user of the training effect. An information processing method comprising the steps of:

2. A computer comprising: extracting bioinformation and a specific movement of the head from a plurality of bioelectric potential measurement results corresponding to time-series information including at least myoelectric information obtained by measuring a bioelectric potential in the head of the human body; displaying the extracted biometric information and the specific head movement on a screen in association with an image relating to the user's head; displaying advice information corresponding to the specific movement on the screen; An image of the user before the user performs the specific movement and an image of the user after the user performs the specific movement are displayed so as to be compared. An information processing method comprising the steps of:

Images