Information processing method, program, and communication device

Abstract

The information processing method executed by the communication device includes acquiring a first biosignal from a first earpiece equipped with a biosensor, acquiring a second biosignal from a second earpiece equipped with a biosensor, dividing the first biosignal and the second biosignal at predetermined time intervals, determining a similarity between first feature data based on the first biosignal for the divided predetermined interval and second feature data based on the second biosignal for a predetermined time shifted in time with respect to the predetermined interval, and performing a synchronization process to synchronize the first biosignal and the second biosignal based on the similarity.

Description

[Technical field]

[0001] The present invention relates to an information processing method, a program, and a communication device. [Background technology]

[0002] Conventionally, wireless earphones equipped with a biosensor are known. For example, Patent Document 1 discloses a technology in which wireless earphones worn on the left and right ears transmit biosignals acquired by the sensor to a cradle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-195179 A Summary of the Invention [Problem to be solved by the invention]

[0004] Here, in the technology described in Patent Document 1, when the wireless earphone transmits a biosignal to the cradle, the cradle processes and displays the left and right biosignals separately. However, in the conventional technology, this is not a problem because the left and right biosignals are processed separately. However, when considering wirelessly transmitting biosignals from the left and right wireless earphones, there is a risk that an error will occur in the timing at which the receiving communication device receives the left and right biosignals due to communication delays, etc. In this case, there is a problem that the data based on the left and right biosignals cannot be processed appropriately.

[0005] The disclosed technology has been made in consideration of these circumstances, and aims to provide technology that enables a communication terminal to properly synchronize left and right biological signals received from wireless earphones. [Means for solving the problem]

[0006] An information processing method that is one aspect of the disclosed technology is an information processing method executed by a communication device, and includes the steps of acquiring a first biosignal from a first earpiece equipped with a biosensor, acquiring a second biosignal from a second earpiece equipped with a biosensor, dividing the first biosignal and the second biosignal at predetermined time intervals, determining a similarity between first feature data based on the first biosignal of the divided predetermined section and second feature data based on the second biosignal of a predetermined time shifted in time based on the first feature data of the predetermined section, and performing a synchronization process to synchronize the first biosignal and the second biosignal based on the similarity.

[0007] A program according to another aspect of the present disclosure includes a first acquisition step of acquiring a first biosignal from a first earpiece having a biosensor, a second acquisition step of acquiring a second biosignal from a second earpiece having a biosensor, a division step of dividing the first biosignal and the second biosignal into predetermined time intervals, a determination step of determining a similarity between first feature data based on the first biosignal of the divided predetermined interval and second feature data based on the second biosignal of a predetermined interval shifted in time based on the first feature data of the predetermined interval, and a synchronization step of synchronizing the first biosignal and the second biosignal based on the similarity.

[0008] A communication device according to another aspect of the present disclosure includes a first acquisition unit that acquires a first biosignal from a first earpiece having a biosensor, a second acquisition unit that acquires a second biosignal from a second earpiece having a biosensor, a division unit that divides the first biosignal and the second biosignal at predetermined time intervals, a determination unit that determines a similarity between first feature data based on the first biosignal of the divided predetermined section and second feature data based on the second biosignal of a predetermined section shifted in time based on the first feature data of the predetermined section, and a synchronization unit that synchronizes the first biosignal and the second biosignal based on the similarity. Effect of the Invention

[0009] According to the present invention, the left and right biological signals received by a communication terminal from wireless earphones can be appropriately synchronized. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram for explaining an overview of the present embodiment. [Diagram 2] FIG. 2 is a block diagram showing the configuration of an earpiece according to the present embodiment. [Diagram 3] FIG. 2 is a diagram showing the external shape of an earpiece according to the present embodiment. [Figure 4] 4 is a cross-sectional view showing the IV-IV cross section of the earpiece according to the present embodiment. FIG. [Diagram 5] 2 is a block diagram showing a configuration of a communication terminal according to the present embodiment. FIG. [Figure 6] FIG. 13 is a graph showing fluctuations in α waves of subject A. [Figure 7] FIG. 1 shows the power spectral density for a given 4-second period when the subject had both eyes closed. [Figure 8] FIG. 7 is a diagram showing a matrix of correlation coefficients between EP-L, EP-R, and EP-Diff and PM-T7, PM-Cz, and PM-T8 shown in FIG. 6. [Figure 9] 4 is a diagram showing an example of a first biological signal and a second biological signal before synchronization processing by a synchronization unit. FIG. [Figure 10] 13 is a diagram showing an example in which a determination unit determines a similarity between a first envelope and a second envelope. FIG. [Figure 11] 11 is a diagram illustrating an example of a synchronization process performed by a synchronization unit. [Figure 12] 13 is a flowchart showing an example of processing of a communication terminal M according to the embodiment. [Figure 13] 5 is a flowchart relating to a process of a feature data extraction unit according to the present embodiment. [Figure 14] FIG. 2 is a diagram illustrating a hardware configuration according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the present embodiment will be described with reference to the accompanying drawings. In order to facilitate understanding of the description, the same components in each drawing are denoted by the same reference numerals as much as possible, and duplicated description will be omitted.

[0012] An overview of this embodiment will be described with reference to FIG. 1. The earphone 2 according to this embodiment includes a first earpiece 2R and a second earpiece 2L. The first earpiece 2R is worn on one ear of a user H. The second earpiece 2L is worn on the other ear of the user H. The first earpiece 2R and the second earpiece 2L are configured to be capable of communicating with a smartphone. The smartphone is an example of a communication terminal M.

[0013] <Earpiece configuration> Next, components of the earphone 2 according to this embodiment will be described with reference to Fig. 2. The earphone 2 includes a first earpiece 2R and a second earpiece 2L. The first earpiece 2R includes, as components, a main sensor 271 (first sensor), a reference sensor 272 (third sensor), a ground sensor 273 (fifth sensor), a first A / D conversion unit 274, and a first transmission unit 275.

[0014] The main sensor 271 is provided at a position where it can acquire the first biological information of the user H as an electrical signal. The arrangement position of the main sensor 271 will be described later. The main sensor 271 outputs the sensed first biological information to the first A / D conversion unit 274.

[0015] The reference sensor 272 is provided at a position where it can acquire the third biological information of the user H as an electrical signal. The arrangement position of the reference sensor 272 will be described later. The reference sensor 272 outputs the sensed third biological information to the first A / D conversion unit 274.

[0016] The first biological information and the third biological information are, for example, information related to the brain waves of the user H. Generally, brain waves include a plurality of types of frequency bands such as alpha waves (8-13 Hz), beta waves (14-25 Hz), theta waves (4-7 Hz), delta waves (1-3 Hz), and gamma waves (30 Hz or higher). Therefore, the first biological information and the third biological information may include brain waves of a plurality of frequency bands.

[0017] The ground sensor 273 is a sensor that obtains ground potential information as an electric signal. The arrangement position of the ground sensor 273 will be described later. The ground sensor 273 outputs the sensed ground potential information to the first A / D conversion unit 274.

[0018] The first A / D conversion unit 274 converts each piece of information from analog information to digital information. The first A / D conversion unit 274 also outputs the converted information to the first transmission unit 275.

[0019] The first transmission unit 275 transmits the first biological information output from the main sensor 271 and the third biological information output from the reference sensor 272, which are acquired from the first A / D conversion unit 274, to the communication terminal M. The first transmission unit 275 may also transmit the ground potential information output from the ground sensor 273, which is acquired from the first A / D conversion unit 274, to the communication terminal M. The first transmission unit 275 may generate first difference information, which is a difference between the first biological information output from the main sensor 271 and the ground potential information output from the ground sensor 273, which are acquired from the first A / D conversion unit 274, and transmit the first difference information to the communication terminal M. Similarly, the first transmission unit 275 may generate third difference information, which is a difference between the third biological information output from the reference sensor 272 and the ground potential information output from the ground sensor 273, which are acquired from the first A / D conversion unit 274, and transmit the third difference information to the communication terminal M.

[0020] In the above-mentioned example, the first transmission unit 275 transmits the first biological information or the first difference information to the communication terminal M through the first channel. Moreover, the first transmission unit 275 transmits the third biological information or the third difference information to the communication terminal M through the third channel. The third channel may be the same as or different from the first channel. Moreover, each of the above-mentioned biological information or ground potential information includes information sampled by the first A / D conversion unit 274.

[0021] The second earpiece 2L includes a main sensor 281 (second sensor), a reference sensor 282 (fourth sensor), a ground sensor 283 (sixth sensor), a second A / D conversion section 284, and a second transmission section 285 as components.

[0022] The main sensor 281 is provided at a position where it can acquire the second biological information of the user H as an electrical signal. The arrangement position of the main sensor 281 will be described later. The main sensor 281 outputs the sensed second biological information to the second A / D conversion unit 284.

[0023] The reference sensor 282 is provided at a position where it can acquire the fourth biological information of the user H as an electrical signal. The arrangement position of the reference sensor 282 will be described later. The reference sensor 282 outputs the sensed fourth biological information to the second A / D conversion unit 284.

[0024] The second biological information and the fourth biological information are, for example, information related to the brain waves of the user H. The second biological information and the fourth biological information may include brain waves in a plurality of frequency bands, similar to the first biological information and the third biological information.

[0025] The ground sensor 283 is a sensor that obtains ground potential information as an electric signal. The arrangement position of the ground sensor 283 will be described later. The ground sensor 283 outputs the sensed ground potential information to the second A / D conversion unit 284.

[0026] The second A / D conversion unit 284 converts each piece of information from analog information to digital information. The second A / D conversion unit 284 also outputs the converted information to the second transmission unit 285.

[0027] The second transmission unit 285 transmits the second biological information output from the main sensor 281 and the fourth biological information output from the reference sensor 282, which are acquired from the second A / D conversion unit 284, to the communication terminal M. The second transmission unit 285 may also transmit the ground potential information output from the ground sensor 283, which is acquired from the second A / D conversion unit 284, to the communication terminal M. The second transmission unit 285 may generate second difference information, which is a difference between the second biological information output from the main sensor 281 and the ground potential information output from the ground sensor 283, which are acquired from the second A / D conversion unit 284, and transmit the second difference information to the communication terminal M. Similarly, the second transmission unit 285 may generate fourth difference information, which is a difference between the fourth biological information output from the reference sensor 282 and the ground potential information output from the ground sensor 283, which are acquired from the second A / D conversion unit 284, and transmit the fourth difference information to the communication terminal M.

[0028] In the above-mentioned example, the second transmission unit 285 transmits the second biological information or the second difference information to the communication terminal M through the second channel. Moreover, the second transmission unit 285 transmits the fourth biological information or the fourth difference information to the communication terminal M through the fourth channel. Moreover, the fourth channel may be the same channel as the second channel or a different channel. Moreover, each of the above-mentioned biological information or ground potential information includes information sampled by the second A / D conversion unit 284.

[0029] Next, an example of the external shape of the earpiece and the arrangement of the sensors will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a diagram showing the external shape of the earpiece, and Fig. 4 is a cross-sectional view showing the earpiece along the line IV-IV. The first earpiece 2R and the second earpiece 2L have the same basic structure and differ only in the shape suitable for insertion into the left and right ears, so the first earpiece 2R will be used as an example for the description.

[0030] The first earpiece 2R includes a housing 21, an ear tip 22, and a wing 23. The housing 21 is a member having a cavity therein, and this cavity contains a speaker 24 and a battery 25. The ear tip 22 is attached to a nozzle 26 protruding from the housing 21.

[0031] The nozzle 26 has a shape such that a flange is formed at one end of a cylindrical member. Specifically, the nozzle 26 has a cylindrical portion 31 that is positioned in the ear canal of the wearer when the first earpiece 2R is attached, a flange portion 32 fixed to the housing 21, and a sound guide portion 33 that connects the interiors of the cylindrical portion 31 and the flange portion 32 in the first direction X.

[0032] The cylindrical portion 31 extends in the first direction X so as to protrude from the housing 21, and a locking protrusion 35 for detachably locking the ear tip 22 is formed on the tip side of the cylindrical portion 31. The flange portion 32 is formed on the base end side of the cylindrical portion 31. The sound guide portion 33 functions as a passage through which sound from the speaker 24 passes. Such a nozzle 26 is formed as a rigid body. As long as it has this characteristic, the material for forming the nozzle 26 is not limited, but one example is hard ABS resin.

[0033] The housing 21 includes a nozzle fixing portion 40 to which the flange portion 32 of the nozzle 26 is fixed, and an expansion portion 42 that extends from the nozzle fixing portion 40 to the opposite side of the nozzle 26 and is more expanded than the nozzle fixing portion 40. The nozzle fixing portion 40 includes an opening 50 through which the cylindrical portion 31 of the nozzle 26 is inserted, a restricting portion 52 that abuts against the flange portion 32 of the nozzle 26 to restrict the nozzle 26 from falling out of the opening 50, and a surrounding portion 54 that connects between the restricting portion 52 and the expansion portion 42 and surrounds the flange portion 32. The nozzle 26 can be fixed to the nozzle fixing portion 40 by, for example, fitting the flange portion 32 into the surrounding portion 54, or by bonding the flange portion 32 to the restricting portion 52 or the surrounding portion 54.

[0034] The extension portion 42 is formed so as to gradually expand as it moves away from the nozzle fixing portion 40. Inside the extension portion 42, the speaker 24 is accommodated on the nozzle 26 side, and the battery 25 is accommodated in a portion that is larger than the portion accommodating the speaker 24. The extension portion 42 also has an opening 60 for wiring at the end 42a where the expansion ends. The speaker 24 and the battery 25 are connected to the board 70 by wiring through this opening 60. The opening 60 is also used to insert the speaker 24 and the battery 25 into the housing 21 when assembling the first earpiece 2R.

[0035] The extension section 42 has an end surface 42b where the extension ends, which is formed as a flat surface. A plate 72 to which a substrate 70 is fixed is placed on the end surface 42b, and a cover 74 that covers the substrate 70 and the plate 72 is attached. The substrate 70 is provided with an antenna for wireless communication. The antenna is compatible with wireless communication standards such as Bluetooth (registered trademark). Therefore, the first earpiece 2R is configured as a wireless earphone, and is wirelessly connected to devices such as a mobile terminal and a laptop, and communicates data such as sound with these devices. The substrate 70 is provided with a first A / D conversion unit 274 and a first transmission unit 275.

[0036] The wing 23 includes an annular attachment portion 80 that is fitted into the circumferential surface of the cover 74, and an earmuff portion 82 that protrudes from the attachment portion 80. The earmuff portion 82 protrudes from the attachment portion 80 in a generally U-shape toward the ear tip 22. The earmuff portion 82 mainly functions to hook onto the outer ear of the wearer when the first earpiece 2R is worn, and supports the first earpiece 2R so that it does not fall off the concha of the wearer. The wing 23, like the housing 21, can be made of a material that is elastic and flexible.

[0037] The ear tip 22 is formed of a first member 22A and a second member 22B having electrical conductivity. For example, the first member 22A and the second member 22B are formed of different materials and are detachable. The shape of the first member 22A is not limited to the example shown in Figs. 3 and 4, and may be any shape as long as it has a portion that contacts the inner wall of the wearer's ear canal and this contact portion is appropriately configured to contact the ear canal. In addition, it is preferable that the surface area of ​​this contact portion is large.

[0038] The ear tip 22 includes a first member 22A located on the eardrum side and a second member 22B located on the housing 21 side. The first member 22A is made of, for example, conductive rubber, and this conductive rubber contains silver or silver chloride. Preferably, to ensure appropriate conductivity, silver or silver chloride is contained in a predetermined mass % or more of the conductive material contained in the conductive rubber.

[0039] The first member 22A may be formed of a silicon material containing a metal-based filler. For example, the first member 22A can be made into a highly conductive material by appropriately mixing silver, copper, gold, aluminum, zinc, nickel, etc., as a metal-based filler into the silicon material. In addition, it is not necessary for all of the filler contained to be silver or silver chloride, as long as a part of the filler is silver or silver chloride. This reduces the content of silver or silver chloride, thereby lowering the hardness of the rubber and making it possible to create a conductive rubber with an appropriate hardness.

[0040] The second member 22B is preferably made of an inexpensive non-conductive elastic body (such as silicone rubber).

[0041] The housing 21 is made of, for example, conductive rubber, and this conductive rubber contains silver or silver chloride. Preferably, in order to ensure appropriate conductivity, silver or silver chloride is contained in a predetermined mass % or more of the conductive material contained in the conductive rubber.

[0042] The housing 21 may be formed of a silicon material containing a metal-based filler. For example, the first member 22A can be made into a highly conductive material by appropriately mixing silver, copper, gold, aluminum, zinc, nickel, etc., as a metal-based filler into a silicon material. In addition, it is not necessary for all of the filler contained to be silver or silver chloride, as long as a part of the filler is silver or silver chloride. This reduces the content of silver or silver chloride, and therefore the hardness of the rubber can be reduced, making it possible to create a conductive rubber with an appropriate hardness.

[0043] The earmuffs 82 are made of, for example, conductive rubber, and this conductive rubber contains silver or silver chloride. Preferably, to ensure appropriate conductivity, the conductive rubber contains silver or silver chloride at a predetermined mass % or more of the conductive material contained therein.

[0044] The earmuffs 82 may be made of a silicon material containing a metal-based filler. For example, the first member 22A can be made of a highly conductive material by appropriately mixing metal-based fillers such as silver, copper, gold, aluminum, zinc, and nickel into the silicon material. In addition, it is not necessary for all of the filler contained to be silver or silver chloride, as long as a part of the filler is silver or silver chloride. This reduces the content of silver or silver chloride, and therefore the hardness of the rubber can be reduced, making it possible to create a conductive rubber with an appropriate hardness.

[0045] By configuring as described above, the first member 22A of the ear tip 22, the housing 21, and the earmuff 82 have conductivity. In the example shown in the present embodiment, the first member 22A is used as the main sensor 271, the housing 21 is used as the ground sensor 273, and the earmuff 82 is used as the reference sensor 272, but this example is not limited to this. In addition, the conductive material of the housing 21, the conductive material of the first member 22A, and the conductive material of the earmuff 82 may be the same or different. In addition, the conductivity of the first member 22A may be higher than the conductivity of the housing 21 and the earmuff 82. Note that the above-mentioned earpiece shows an example of the disclosed technology, and the disclosed technology is applicable to at least two biological information measuring devices that can acquire biological information and communicate independently.

[0046] 5 is a block diagram showing the configuration of a communication terminal M according to this embodiment. The communication terminal M includes an information processing unit 251 and a storage unit 261.

[0047] The information processing unit 251 includes a first acquisition unit 252, a second acquisition unit 253, a division unit 254, a feature data extraction unit 255, a determination unit 259, and a synchronization unit 260, for example.

[0048] The storage unit 261 stores, for example, a first biological signal 262, a second biological signal 263, a first feature data 264, a second feature data 265, and a correction value 266. The storage unit 261 corresponds to the memory 330 shown in FIG.

[0049] The first biological signal 262 is a biological signal of the user H acquired from the first earpiece 2R. For example, the first biological signal 262 includes at least one of the first biological information, the third biological information, the ground potential information, the first difference information, and the third difference information.

[0050] The second biological signal 263 is a biological signal of the user H acquired from the second earpiece 2L. For example, the second biological signal 263 includes at least one of the second biological information, the fourth biological information, the ground potential information, the second difference information, and the fourth difference information.

[0051] The first feature data 264 is feature data related to the first biological signal 262. The first feature data 264 includes, for example, feature data of a time component or feature data of a frequency component of the first biological signal 262. As described later, the first feature data 264 may include at least one of a peak value of the first biological signal 262, a first power spectrum after frequency conversion of the first biological signal 262, a first power spectrum of a predetermined frequency band, and a first envelope based on the first power spectrum.

[0052] The second feature data 265 is feature data related to the second biological signal 263. The second feature data 265 includes, for example, feature data of a time component or feature data of a frequency component of the second biological signal 263. As described later, the second feature data 265 may include at least one of a peak value, a second power spectrum after frequency conversion of the second biological signal 263, a second power spectrum in a predetermined frequency band, and a second envelope based on the second power spectrum.

[0053] The correction value 266 is a correction value used when synchronizing the first biological signal 262 and the second biological signal 263. The correction value 266 includes a shift value used to shift at least one of the biological signals, and will be described in detail later.

[0054] The first acquisition unit 252 acquires the first biological signal 262 from the first earpiece 2R. For example, the first acquisition unit 252 acquires the first biological signal 262 from the communication terminal M by communication using Bluetooth (registered trademark). The communication using Bluetooth (registered trademark) is realized by the network communication interface 320 shown in FIG.

[0055] The second acquisition unit 253 acquires the second biological signal 263 from the second earpiece 2L. For example, the second acquisition unit 253 acquires the second biological signal 263 from the communication terminal M by communication using Bluetooth (registered trademark).

[0056] The dividing unit 254 divides the biosignal sequentially acquired by the first acquiring unit 252 or the second acquiring unit 253 into a predetermined time interval. For example, when the predetermined time is 10 seconds, the dividing unit 254 divides the biosignal sequentially acquired by the first acquiring unit 252 or the second acquiring unit 253 into a 10 second interval. Note that the predetermined time interval for division can be set arbitrarily.

[0057] The determination unit 259 determines the similarity between first feature data 264 based on the first biological signal 262 of the divided predetermined time (also referred to as a predetermined section) and second feature data 265 based on the second biological signal 263 of a predetermined time shifted in sequence in time with reference to the first feature data of the predetermined section. An example of the processing by the determination unit 259 will be described later with reference to Figures 9 and 10. For the shift, for example, the predetermined section is shifted by a set time from -2 seconds to +2 seconds.

[0058] The synchronization unit 260 synchronizes the first biosignal 262 and the second biosignal 263 based on the similarity between the first feature data 264 and the second feature data 265 determined by the determination unit 259. For example, the synchronization unit 260 uses a time shift amount (correction value) between the first biosignal 262 and the second biosignal 263 obtained from the point in time when the correlation value, which is one of the indexes of similarity determined by the determination unit 259, is highest, to shift one of the biosignals in time by the correction value, thereby synchronizing the first biosignal 262 and the second biosignal 263. In this way, the left and right biosignals received by the communication terminal M from the earpieces can be appropriately synchronized.

[0059] The communication terminal M may include a feature data extraction unit 255. The feature data extraction unit 255 extracts first feature data 264 and second feature data 265 from the first biometric signal 262 and the second biometric signal 263, respectively.

[0060] The feature data extraction unit 255 may include, for example, a frequency conversion unit 256. The frequency conversion unit 256 performs frequency conversion on the first biological signal 262 and the second biological signal 263. The frequency conversion unit 256 applies FFT (Fast Fourier Transform) processing, wavelet transformation, or the like to the first biological signal 262 and the second biological signal 263 including brain waves of a plurality of frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves, and calculates frequency components (e.g., power spectrum).

[0061] When the feature data extraction unit 255 includes the frequency conversion unit 256, the determination unit 259 may determine the similarity between a first power spectrum after frequency conversion of the first biological signal 262 included in the first feature data 264 and a second power spectrum after frequency conversion of the second biological signal 263 included in the second feature data 265. The determination unit 259 may determine the similarity between, for example, the first power spectrum of a predetermined section of frequency components and the second power spectrum of a predetermined section of frequency components shifted by a set time. As described above, by using frequency components in which features are likely to appear as feature data, it is possible to easily determine the similarity.

[0062] The feature data extraction unit 255 may further include, for example, a frequency band extraction unit 257. The frequency band extraction unit 257 extracts a predetermined frequency band after frequency conversion. As a preferred example, the frequency band extraction unit 257 applies a band pass filter to the frequency components calculated by the frequency conversion unit 256 to extract the power spectrum of alpha waves.

[0063] When the feature data extraction unit 255 includes the frequency band extraction unit 257, the determination unit 259 may determine the similarity using a first power spectrum of a predetermined frequency band among the frequency components for a predetermined section, and a second power spectrum of the predetermined frequency band among the frequency components for a predetermined section shifted by a set time. The determination unit 259 may determine the similarity between the first power spectrum of α waves and the second power spectrum of α waves, for example. As described above, it is possible to set a predetermined frequency band in which a wavy shape that can be used as a feature is likely to be formed, and determine the similarity based on the power spectrum of this predetermined frequency band.

[0064] The feature data extraction unit 255 may further include, for example, an envelope calculation unit 258. The envelope calculation unit 258 calculates a first envelope based on the first power spectrum and a second envelope based on the second power spectrum. For example, the envelope calculation unit 258 may apply a Hilbert transform to the first power spectrum and the second power spectrum of a plurality of frequency bands to calculate the first envelope and the second envelope. The envelope calculation unit 258 may also apply a Hilbert transform to the first power spectrum and the second power spectrum of a predetermined frequency band to calculate the first envelope and the second envelope. As a preferred example, the envelope calculation unit 258 applies a Hilbert transform to the first power spectrum of α waves and the second power spectrum of α waves to calculate the first envelope and the second envelope. By performing the Hilbert transform, the first envelope and the second envelope can be calculated that are not dependent on the difference in power value between the first power spectrum and the second power spectrum.

[0065] When feature data extraction section 255 includes envelope calculation section 258, determination section 259 may determine the similarity between a first envelope included in first feature data 264 for a predetermined section and a second envelope included in second feature data 265 for a predetermined section shifted by a set time. For example, determination section 259 may determine, for each predetermined section, the time point at which the correlation value between the first envelope and the second envelope shifted by the set time is highest. As described above, by using an envelope that is not based on the difference in power values ​​between the left and right as feature data, a more appropriate similarity determination can be performed.

[0066] The above-mentioned determination unit 259 may determine the similarity between the first feature data based on the first biosignal of the divided predetermined section and the second feature data based on the second biosignal of a predetermined time shifted in sequence with respect to the predetermined section of the first feature data. For example, when the division unit 254 divides the first biosignal 262 and the second biosignal 263 at 10-second intervals, the determination unit 259 shifts the second biosignal 263 of the 10-second period corresponding to the first biosignal 262 of the predetermined section in time with respect to the first biosignal 262 of the predetermined section. The determination unit 259 calculates a correlation value between the first feature data 264 of the first biosignal 262 of the predetermined section and the second feature data 265 of the section in which the second biosignal 263 is shifted in time. As a result, an objective correlation value is used as the similarity between the first feature data 264 and the second feature data 265, so that the accuracy of similarity determination can be improved and the synchronization process between the first biosignal and the second biosignal can be more appropriately performed.

[0067] The above-mentioned determination unit 259 may calculate a similarity between an amplitude value indicating the first feature data 264 and an amplitude value indicating the second feature data 265. The determination unit 259 may calculate a similarity between an amplitude value of a first power spectrum in a predetermined section and an amplitude value of a second power spectrum in a predetermined section shifted by a set time, for example. This makes it possible to calculate a similarity using the amplitude value, which makes it easier to determine the similarity and reduces the processing load, thereby enabling synchronization processing between the first biological signal 262 and the second biological signal 263.

[0068] Here, we will explain the details of an experiment conducted by the inventors to confirm whether the synchronization processing of the disclosed technology works effectively on the biosignals obtained from each earpiece equipped with a biosensor, using alpha waves contained in electroencephalograms.

[0069] <Experiment Overview> (subject) This experiment was conducted on three subjects, A to C. First, this experiment will be described.

[0070] (obtained items) The subject's electroencephalogram signal, which serves as a reference value, is acquired using an electroencephalogram meter, Polymate Mini (registered trademark), manufactured by Miyuki Giken Co., Ltd. In addition, the subject's electroencephalogram signal is acquired from both ears using earpieces similar to the first earpiece 2R and the second earpiece 2L.

[0071] (How to obtain) For the same subject, electrodes were attached to the subject's head at T7 (left middle temporal region), Cz (median center), and T8 (right middle temporal region) according to the international 10 / 20 method using the above-mentioned electroencephalograph, and the above-mentioned earpieces were attached to the subject's left and right ears. The subject repeatedly opened and closed both eyes every 15 seconds, and electroencephalogram signals were obtained from the above-mentioned electroencephalograph and earpieces for at least 120 seconds (4 cycles, assuming that opening and closing is one cycle).

[0072] (process) Each electroencephalogram signal acquired by the electroencephalogram meter and the electroencephalogram signal acquired by the left earpiece are transmitted to a first personal computer. The electroencephalogram signal acquired by the right earpiece is transmitted to a second personal computer different from the first personal computer. The second personal computer transmits the electroencephalogram signal received from the right earpiece to the first personal computer. The first personal computer executes the above-mentioned synchronization process for each electroencephalogram signal. The first personal computer generates a difference signal between the electroencephalogram signal acquired by the left earpiece and the electroencephalogram signal acquired by the right earpiece after the synchronization process. In addition, frequency conversion and a bandpass filter are applied to each electroencephalogram signal including the difference signal, the left and right electroencephalogram signals, and the electroencephalogram signal acquired by the electroencephalogram meter, and the power spectrum of alpha waves is extracted.

[0073] (Evaluation Index) In this experiment, the indices used to evaluate the similarity of each extracted EEG signal were the alpha wave power obtained from the left ear EEG signal, the alpha wave power obtained from the right ear EEG signal, and the correlation coefficients (correlation values) between the alpha wave power of the differential signal between the left ear EEG signal and the right ear EEG signal, and the alpha wave power obtained from the EEG meter.

[0074] (Example of Subject A) An example of data of subject A in this experiment will be described. FIG. 6 is a diagram showing the fluctuation of the power of alpha waves of subject A. FIG. 6 is a diagram showing the 4-second time window for each of the EEG signals acquired by the electroencephalograph and the earpiece after the above-mentioned synchronization processing, in which the alpha wave power is calculated and plotted in chronological order by applying frequency conversion and bandpass filtering to the EEG signals of each time window. For example, the power shown in FIG. 6 indicates the peak value of the alpha wave power. EP-L, EP-R, and EP-Diff respectively represent the power of alpha waves obtained from the EEG signal of the left ear, the power of alpha waves obtained from the EEG signal of the right ear, and the power of alpha waves of the difference signal between the EEG signal of the left ear and the EEG signal of the right ear. It should be noted that EP-Diff is the difference signal after synchronization processing between the EEG signal of the left ear and the EEG signal of the right ear. PM-T7, PM-Cz, and PM-T8 represent the power of alpha waves obtained from the electroencephalographs attached to T7, Cz, and T8, respectively.

[0075] In the example shown in Figure 6, it can be seen that the power of alpha waves in each EEG signal also tends to decrease when the subject has both eyes open, and tends to increase when both eyes are closed. Alpha waves are said to occur in a relaxed state, and it is believed that by relaxing with the eyes closed, more alpha waves are generated than when the eyes are open. This means that each EEG signal can be said to be appropriately extracted from each device.

[0076] Fig. 7 shows the power spectral density for a given 4-second period when the subject had both eyes closed. EP-L, EP-R, and EP-Diff in Fig. 7 show the power spectral density of the EEG signal at the left ear, the power spectral density of the EEG signal at the right ear, and the power spectral density of the differential signal between the EEG signals at the left ear and the right ear, respectively. PM-T7, PM-Cz, and PM-T8 show the power spectral density of the EEG signal obtained from the EEG monitors attached to T7, Cz, and T8, respectively.

[0077] In the example shown in Fig. 7, the power spectrum density of alpha waves, which is a frequency band of 8 to 13 Hz, peaks at around 10 Hz for all EEG signals. This indicates that both the EEG and the earpieces can adequately extract the characteristics of the subject's brain state (e.g., alpha waves). It can also be said that the earpieces can extract the peak value of alpha waves in the same way as the EEG.

[0078] FIG. 8 is a diagram showing a matrix of correlation coefficients between EP-L, EP-R, and EP-Diff shown in FIG. 6 and PM-T7, PM-Cz, and PM-T8. First, the electroencephalograph acquires an electroencephalogram signal from the scalp at a predetermined position on the head, whereas the earpiece acquires an electroencephalogram signal from the ear. That is, the acquisition positions of the electroencephalogram signal are different for each device. This experiment was conducted under the assumption that the electroencephalogram signal acquired by the electroencephalograph and the electroencephalogram signal acquired from the earpiece do not have exactly the same waveform due to differences in the acquisition positions of the electroencephalogram signal and differences in the acquisition devices. Therefore, in this experiment, if the correlation coefficient is 0.3 or more, it can be evaluated that the similarity between the power value of the alpha wave acquired from the electroencephalograph and the power value of the alpha wave acquired from the earpiece is high.

[0079] In the example shown in Figure 8, all correlation coefficients are greater than or equal to 0.3, which indicates that there is a high degree of similarity between the alpha wave power value obtained from the electroencephalograph and the alpha wave power value obtained from the earpiece.

[0080] The above experimental results show that the earpieces equipped with biosensors can appropriately acquire alpha waves contained in electroencephalograms. In addition, the synchronization process of the disclosed technology works effectively with the biosignals acquired from the earpieces equipped with biosensors using the alpha waves contained in electroencephalograms.

[0081] <Example of judgment and synchronization process> Next, an example of the processing of the determination section 259 and the synchronization section 260 will be described with reference to Figs. 9 to 11. Fig. 9 is a diagram showing an example of the first biological signal 262 and the second biological signal 263 before synchronization processing by the synchronization section 260. In the example shown in Fig. 9, the solid line R represents the electroencephalogram signal itself for a predetermined one second acquired from the first earpiece 2R, and the dashed line L represents the electroencephalogram signal itself for one second acquired from the second earpiece 2L in the same time period. In the example shown in Fig. 9, sampling is performed at 600 Hz, and there are sampling values ​​of 1 second÷(1 / 600)=600 data points.

[0082] Fig. 10 is a diagram showing an example in which the determination unit 259 determines the similarity between the first envelope of the first biosignal 262 shown in Fig. 9 and the second envelope of the second biosignal 263. In the example shown in Fig. 10, the determination unit 259 calculates correlation values ​​between the first envelope and each of the second envelopes of the second biosignal 263 for one second, which are shifted by one data point (e.g., 1 / 600 seconds) from -2.5 seconds to +2.5 seconds based on the first envelope, and shows a graph showing the fluctuation of the correlation values. In the example shown in Fig. 10, the correlation value becomes highest by shifting the second biosignal 263 forward in time by 254 data points.

[0083] Fig. 11 is a diagram showing an example of synchronization processing by synchronization section 260. In the example shown in Fig. 11, synchronization section 260 shifts the one-second electroencephalogram signal acquired from second earpiece 2L forward in time by 254 data points in accordance with the highest correlation value among the correlation values ​​calculated by determination section 259. This makes it possible to synchronize the one-second electroencephalogram signal acquired from first earpiece 2R with the one-second electroencephalogram signal acquired from second earpiece 2L in the same time period.

[0084] <Operation process> Next, an operation according to the embodiment will be described with reference to Fig. 12 and Fig. 13. Fig. 12 is a flowchart showing an example of a process of the communication terminal M according to the embodiment.

[0085] In step S11, the first acquirer 252 acquires the first biosignal 262 from the first earpiece 2R that is equipped with a biosensor.

[0086] In step S12, the second acquisition section 253 acquires a second biological signal 263 from the second earpiece 2L that is equipped with a biological sensor.

[0087] In step S13, when a predetermined time has elapsed since acquiring the first biological signal 262 or the second biological signal 263, the dividing unit 254 divides the first biological signal 262 and the second biological signal 263 at predetermined time intervals.

[0088] In step S14, the feature data extraction unit 255 extracts first feature data 264 based on the first biosignal 262 in the divided predetermined section and second feature data 265 based on the second biosignal 263 in the divided predetermined section.

[0089] In step S15, the judgment unit 259 judges the similarity between first feature data 264 based on the first biological signal 262 of the divided specified section and second feature data 265 based on the second biological signal 263 of a specified time shifted in time based on the specified section.

[0090] In step S16, the synchronization unit 260 synchronizes the first biological signal 262 and the second biological signal 263 based on the similarity.

[0091] A detailed example of the processing of the feature data extraction unit 255 will be described with reference to Fig. 13. Fig. 13 is a flowchart related to the processing of the feature data extraction unit 255 according to this embodiment. In the example shown in the figure, in step S21, the frequency conversion unit 256 performs frequency conversion on the first biological signal 262 and the second biological signal 263.

[0092] In step S22, the frequency band extraction unit 257 extracts a predetermined frequency band after the frequency conversion.

[0093] In step S23, the envelope calculation unit 258 applies the Hilbert transform to the first power spectrum to calculate the first envelope, and applies the Hilbert transform to the second power spectrum to calculate the second envelope. After that, in step S15 shown in Fig. 12, similarity is determined using each envelope as feature data.

[0094] <Modification> The above-mentioned embodiment or each example is for facilitating understanding of the present invention, and is not to be construed as limiting the present invention. The present invention may be modified / improved without departing from the spirit thereof, and the present invention also includes equivalents thereof. In addition, the present invention can form various disclosures by appropriate combinations of multiple components disclosed in the above-mentioned embodiment or each example. For example, some components may be deleted from all components shown in the embodiment. Furthermore, components may be appropriately combined in different embodiments.

[0095] The feature data extraction unit 255 may include only the frequency conversion unit 256 and the envelope calculation unit 258, and may not include the frequency band extraction unit 257. In this case, the envelope calculation unit 258 calculates the first envelope and the second envelope based on the first power spectrum and the second power spectrum of the frequency components calculated by the frequency conversion unit 256. The envelope calculation unit 258 calculates the first envelope and the second envelope by applying a Hilbert transform to the first power spectrum and the second power spectrum of the frequency components including a plurality of frequency bands such as α waves, β waves, θ waves, δ waves, and γ waves calculated by the frequency conversion unit 256, for example.

[0096] In the above embodiment, the peak value of the biosignal, the power spectrum after frequency conversion of the biosignal, the power spectrum of a predetermined frequency band, and the envelope based on the power spectrum are given as examples of the characteristic data of the biosignal, but the present invention is not limited thereto. The characteristic data of the biosignal may be a characteristic noise contained in the biosignal. By aligning the generation position of this noise, it is possible to synchronize at least two biosignals.

[0097] In the above embodiment, the frequency band extraction unit 257 extracts the power spectrum of α waves, but the present invention is not limited to this. The frequency band extraction unit 257 may extract the power spectrum of frequency bands other than α waves, such as β waves, θ waves, δ waves, and γ waves.

[0098] Furthermore, the synchronization unit 260 described above may apply the correction value 266 used in the synchronization process of a predetermined section to another section to perform the synchronization process. For example, the synchronization unit 260 may apply the correction value 266 used in the synchronization process of a certain section to the synchronization process of a subsequent predetermined section. This makes it possible to omit processes such as extraction of feature data and determination of similarity for the predetermined section, thereby reducing the processing load of the synchronization process. When the predetermined section has elapsed, extraction of feature data and the like are performed again.

[0099] The communication terminal M may include a post-processing unit that executes a predetermined process on the first biosignal 262 and the second biosignal 263 after the synchronization process. The post-processing unit may, for example, calculate a differential signal between the first biosignal 262 and the second biosignal 263. This makes it possible to cancel common noises contained in the first biosignal 262 and the second biosignal 263, and to acquire a biosignal of good quality. The post-processing unit may also perform cross-reference processing to increase signal components or increase potential differences by performing cross-processing such as processing the first bioinformation and the fourth bioinformation as main bioinformation and reference bioinformation, respectively, and processing the second bioinformation and the third bioinformation as main bioinformation and reference bioinformation, respectively.

[0100] The post-processing unit may extract, for example, brain waves of a predetermined frequency band from the biological signal after the synchronization process. Specifically, the post-processing unit may check, for example, whether the user H is in a relaxed state by extracting α waves. The post-processing unit may check, for example, whether the user H is in an irritated state by extracting β waves. The post-processing unit may check, for example, whether the user H is in a light sleep state by extracting θ waves. The post-processing unit may check, for example, whether the user H is in a deep sleep state by extracting δ waves. The post-processing unit may check, for example, whether the user H is in a focused state by extracting γ waves. The post-processing unit may extract a frequency band different from the predetermined frequency band used when extracting the feature data according to the user's settings, etc.

[0101] <Hardware configuration> 14 is a block diagram showing a hardware configuration according to this embodiment. The communication terminal M is configured by an information processing device, and is, for example, a mobile terminal (such as a smartphone), a computer, a tablet terminal, etc. The communication terminal M is also referred to as a communication terminal 300.

[0102] The communications terminal 300 includes one or more processors (eg, CPUs) 310, one or more network communications interfaces 320, a memory 330, a user interface 350, and one or more communications buses 370 for interconnecting these components.

[0103] The user interface 350 includes a display 351 and an input device (such as a keyboard and / or a mouse or any other pointing device) 352. The user interface 350 may also be a touch panel.

[0104] Memory 330 may be, for example, a high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state storage device, or may be a non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

[0105] Another example of memory 330 may be one or more storage devices located remotely from processor 310. In one embodiment, memory 330 stores the following programs, modules, and data structures, or a subset thereof: Memory 330 may also be a computer-readable non-transitory storage medium.

[0106] The one or more processors 310 read and execute a program from the memory 330 as necessary. For example, the one or more processors 310 execute a program stored in the memory 330 to perform the processing of the information processing unit 251. [Explanation of symbols]

[0107] 2. Earphones 2R 1st earpiece 2L 2nd earpiece 22 Ear tips 251 Information Processing Department 252 First acquisition part 253 Second Acquisition Department 254 Division 255 Feature Data Extraction Unit 256 Frequency conversion section 257 Frequency band extraction section 258 Envelope Calculation Unit 259 Judgment section 260 Synchronization section 261 Storage section 271, 281 Main sensor 272, 282 Reference Sensor 273, 283 Ground Sensor 274 1st A / D conversion section 275 First Transmission Unit 284 Second A / D conversion section 285 Second Transmission Unit 300 Communication terminal 310 Processor 320 Network Communication Interface 330 Memory H user M Communication Terminal

Claims

1. An information processing method executed by a communication device, comprising: Obtaining a first biosignal from a first earpiece comprising a biosensor; acquiring a second biosignal from a second earpiece comprising a biosensor; Dividing the first biological signal and the second biological signal at predetermined time intervals; performing frequency conversion on the first biological signal and the second biological signal for each predetermined time; determining a similarity between first feature data including a first power spectrum after frequency conversion of a first biological signal in a divided predetermined section and second feature data including a second power spectrum after frequency conversion of a second biological signal in the predetermined time shifted in time with respect to the first feature data in the predetermined section; An information processing method that performs a synchronization process to synchronize the first biological signal and the second biological signal based on the similarity.

2. The method further comprises extracting a predetermined frequency band after the frequency conversion; The determining step comprises: The information processing method according to claim 1 , further comprising determining the similarity using a first power spectrum and a second power spectrum of the predetermined frequency band.

3. calculating a first envelope based on the first power spectrum and a second envelope based on the second power spectrum; The determining step comprises: The information processing method according to claim 2 , further comprising determining a similarity between the first envelope included in the first feature data and the second envelope included in the second feature data.

4. The determining step comprises: The information processing method according to claim 1 , further comprising: calculating a correlation value between the first feature data and the second feature data.

5. The synchronizing step includes: The information processing method according to claim 1 , further comprising: applying a correction value used in the synchronization process for the predetermined section to another section to perform the synchronization process.

6. The information processing method according to claim 1 , further comprising: generating a differential signal using the first biological signal and the second biological signal after the synchronization process.

7. For communication devices A first acquisition step of acquiring a first biosignal from a first earpiece including a biosensor; a second acquisition step of acquiring a second biosignal from a second earpiece including a biosensor; A dividing step of dividing the first biological signal and the second biological signal into predetermined time intervals; performing frequency conversion on the first biological signal and the second biological signal at each predetermined time; a determination step of determining a similarity between first feature data including a first power spectrum after frequency conversion of a first biological signal in a divided predetermined section and second feature data including a second power spectrum after frequency conversion of a second biological signal in the predetermined time shifted in time with respect to the first feature data in the predetermined section; A synchronization step of synchronizing the first biological signal and the second biological signal based on the similarity.

8. a first acquisition unit that acquires a first biosignal from a first earpiece that includes a biosensor; a second acquisition unit that acquires a second biological signal from a second earpiece that includes a biological sensor; A division unit that divides the first biological signal and the second biological signal at predetermined time intervals; a frequency conversion unit that performs frequency conversion on the first biological signal and the second biological signal at each predetermined time; a determination unit that determines a similarity between first feature data including a first power spectrum after frequency conversion of a first biological signal in a divided predetermined section and second feature data including a second power spectrum after frequency conversion of a second biological signal in the predetermined time shifted in time with respect to the first feature data in the predetermined section; A communication device comprising: a synchronization unit that synchronizes the first biological signal and the second biological signal based on the similarity.

Images