Real-time in-ear electroencephalogram signal verification
By placing in-ear electrodes and speakers in wearable in-ear devices, real-time calibration and verification of EEG signals is solved, and the problem of EEG systems in the prior art is difficult to portable, real-time calibration and verification of signal quality in non-medical environments, and the problem of EEG systems in real-time is realized. Accurate neural signal data acquisition and error detection in noisy environments.
Patent Information
- Application Number
- CN202380032976.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-06
- Filing Date
- 2023-04-04
- Publication Date
- 2024-11-26
Smart Images

Figure CN119031882A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to electroencephalography (EEG), and in particular to real-time in-ear EEG signal verification in wearable devices. Background Art
[0002] Electroencephalogram (EEG) is a method for recording an electrogram of the electrical activity of a user's brain. More specifically, EEG measures electrical signal voltage fluctuations caused by electrical activity within a large number of neurons in the brain using one or more electrodes placed in contact with the user's anatomical structure (e.g., scalp) to generate EEG signal data (e.g., neural signal data, brain signal data). Common EEG analysis techniques include evoked potentials (EP), which involve averaging EEG activity that is time-locked to the presentation of stimuli (e.g., visual stimuli, somatosensory stimuli, or auditory stimuli (e.g., one or more clicks, one or more chirps and pure tones, etc.)). EP technology may be useful in collecting EEG signal data, which can help measure what a user is focusing on (e.g., trying to listen to) in real time (e.g., in a crowded restaurant) and improve the user's auditory perception. However, EEG data is traditionally collected using medical or hospital-grade EEG monitoring equipment, which places several electrodes on the scalp, allowing the brain's electrical activity to be monitored from multiple angles. Such EEG systems are not suitable for use in a portable, real-time manner outside of an outpatient setting. Summary of the invention
[0003] Embodiments include an in-ear device (IED) of a wearable system that can collect electrical signals corresponding to a user's brain activity in response to stimulation, wherein the electrical signals can be used to generate EEG signal data that can inform the user what the user is paying attention to, and wherein the wearable system is also capable of performing real-time in-ear EEG signal verification to calibrate the EEG signal data and confirm the accuracy of the data.
[0004] According to a first aspect of the present disclosure, a real-time in-ear EEG signal verification system is provided, the system comprising: an in-ear device (IED), the IED being configured to be placed in an ear canal of a user, the IED comprising a speaker and an in-ear electrode, the speaker being configured to present a calibration audio signal to the user, wherein the calibration audio signal is embedded with a predetermined audible feature, the in-ear electrode being configured to contact an inner surface of the ear canal; and a controller being configured to: instruct the speaker to present the calibration audio signal to the user; generate neural signal data based on electrical signals from the in-ear electrodes, the electrical signals corresponding to the user's brain activity in response to the predetermined audible feature; determine the quality of the generated neural signal data; and perform actions based on the quality of the neural signal data.
[0005] In some embodiments, the electrical signal is an electroencephalogram (EEG) signal, and wherein the neural signal data is EEG signal data.
[0006] In some embodiments, the system further comprises a notification unit, wherein the controller is configured to determine the quality of the neural signal data by determining whether the neural signal data satisfies a predetermined amplitude suppression condition and a predetermined time condition after a predetermined audible feature begins, and wherein the controller is configured to, in response to determining that the neural signal data satisfies the predetermined amplitude suppression condition and the predetermined time condition, instruct the notification unit to notify the user that the in-ear electrode is in contact with the inner surface of the ear canal.
[0007] In some embodiments, the controller is further configured to, in response to determining that the neural signal data does not satisfy the predetermined amplitude suppression condition and the predetermined time condition, determine a result of the pitch discrimination operation.
[0008] In some embodiments, the controller is further configured to, in response to determining that the result of the pitch recognition operation is positive, instruct the notification unit to notify the user of one of the following:
[0009] In some embodiments, the in-ear electrode is not in contact with the inner surface of the ear canal; or the in-ear electrode is malfunctioning.
[0010] In some embodiments, the controller is further configured to, in response to determining that the result of the pitch discrimination operation is negative, instruct the notification unit to notify the user of the condition of the hearing loss.
[0011] In some embodiments, the controller is further configured to perform signal processing to correct a gain of the audio signal to compensate for the hearing loss.
[0012] In some embodiments, the predetermined audible characteristic comprises one or more click sounds, and wherein the calibration audio signal comprises a piece of music synthesized with the plurality of click sounds.
[0013] In some embodiments, the method also includes a head-mounted device, the head-mounted device includes the controller, and the head-mounted device also includes at least a second electrode, the second electrode is configured to collect a second electrical signal, the second electrical signal corresponds to the user's brain activity in response to a predetermined audible feature, wherein the controller is configured to generate neural signal data based on the electrical signal from the in-ear electrode and the second electrical signal from the second electrode.
[0014] According to a second aspect of the present disclosure, a method is provided, the method comprising: determining that a user is wearing an in-ear device (IED) by placing the IED in the ear canal of the user, wherein the IED comprises a speaker and an in-ear electrode, the in-ear electrode being configured to contact an inner surface of the ear canal; presenting a calibration audio signal to the user using the speaker, the calibration audio signal being embedded with predetermined audible features; generating electroencephalogram (EEG) signal data based on electrical signals from the in-ear electrode, the electrical signals corresponding to the user's brain activity in response to the predetermined audible features; analyzing the generated EEG signal data, and performing an action based on the analysis.
[0015] In some embodiments, analyzing the EEG signal data includes determining whether the EEG signal data satisfies a predetermined amplitude suppression condition and a predetermined time condition after a predetermined auditory feature begins, and wherein the method further includes: in response to determining that the EEG signal data satisfies the predetermined amplitude suppression condition and the predetermined time condition, notifying the user that the in-ear electrode is in contact with the inner surface of the ear canal.
[0016] In some embodiments, the method further comprises, in response to determining that the EEG signal data does not satisfy the predetermined amplitude suppression condition and the predetermined time range, determining a result of the pitch discrimination operation.
[0017] In some embodiments, the method further includes: based on a positive result of the pitch recognition operation, performing one of the following: notifying the user that the in-ear electrode is not in contact with the inner surface of the ear canal; or notifying the user that the in-ear electrode is malfunctioning.
[0018] In some embodiments, the method further comprises: based on a negative result of the pitch recognition operation: notifying the user of a hearing loss; and correcting a gain of the audio signal to compensate for the hearing loss.
[0019] In some embodiments, the predetermined audible characteristic comprises one or more click sounds, and wherein the calibration audio signal comprises a piece of music synthesized with the plurality of click sounds.
[0020] In some embodiments, the EEG signal data is generated based on the electrical signal from the in-ear electrode and also based on a second electrical signal, the second electrical signal corresponding to the user's brain activity that is time-locked to and responsive to a predetermined audible feature, and the second electrical signal is collected by a second electrode included in a head-mounted device that the user can wear.
[0021] According to a third aspect of the present disclosure, a non-transitory computer-readable medium is provided, which has a computer executable program stored thereon, the program including instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations: determine that a user is wearing an in-ear device (IED) by placing the IED in the ear canal of the user, wherein the IED includes a speaker and an in-ear electrode, the in-ear electrode being configured to contact the inner surface of the ear canal; control the speaker to present a calibration audio signal to the user, the calibration audio signal being embedded with a predetermined audible feature; generate neural signal data based on electrical signals from the in-ear electrode, the electrical signals corresponding to the user's brain activity in response to the predetermined audible feature; determine the quality of the neural signal data; and perform an action based on the determined quality of the neural signal data.
[0022] In some embodiments, the instructions that cause one or more processors to determine the quality of neural signal data include instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: determine whether the neural signal data satisfies a predetermined amplitude suppression condition and a predetermined time condition after a predetermined audible feature begins, and wherein the program also includes instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: in response to determining that the neural signal data satisfies the predetermined condition, controlling a notification unit to notify a user that the in-ear electrode is in contact with the inner surface of the ear canal.
[0023] In some embodiments, the predetermined audible characteristic comprises one or more click sounds, and wherein the calibration audio signal comprises a piece of music synthesized with the plurality of click sounds.
[0024] In some embodiments, the instructions that cause one or more processors to generate neural signal data include instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: generate neural signal data based on electrical signals from in-ear electrodes and also based on a second electrical signal, the second electrical signal corresponding to the user's brain activity in response to a predetermined audible feature, and the second electrical signal is collected by a second electrode, and the second electrode is included in a head-mounted device that the user can wear.
[0025] It should be appreciated that any feature described herein as being suitable for incorporation into one or more aspects or embodiments of the present disclosure is intended to be universal in any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure may be understood by those skilled in the art based on the specification, claims and drawings of the present disclosure. The above general description and the following detailed description are exemplary and illustrative only and are not limitations on the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 is a block diagram of an EEG calibration system according to one or more embodiments.
[0027] Figure 2A is a stereoscopic diagram of a head mounted device (headset) implemented as an eyeglass device according to one or more embodiments.
[0028] Figure 2B is a perspective view of a head mounted device implemented as a head mounted display according to one or more embodiments.
[0029] Figure 3 is a block diagram of an audio system according to one or more embodiments.
[0030] Figure 4 is a flow chart of a method for calibrating neural signal data according to one or more embodiments.
[0031] Figure 5 is a flow chart of a method for determining different actions to perform based on analysis of neural signal data in accordance with one or more embodiments.
[0032] These drawings depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein. DETAILED DESCRIPTION
[0033] The present disclosure relates to performing real-time in-ear EEG signal validation in a wearable system. The technology disclosed herein is intended to provide a wearable system including one or two in-ear devices (IEDs) configured to be worn in the ears of a user - for example, to collect electrical signals from the user's brain. The collected electrical signals can then be processed by a controller (of the IED or a device disposed separately from the IED) to generate EEG signal data (e.g., neural signal data) from stimulus inputs. The wearable system is also capable of determining the validity of the EEG signal data by performing a calibration operation, in which the EEG signal data is analyzed (e.g., to determine the quality of the data) and one or more actions are performed based on the analysis. In fact, by performing the calibration operation, the wearable system is able to confirm in real time whether the generated EEG signal data accurately represents the user's brain activity. Thus, a user of the wearable system can verify for themselves the presence (or absence) of real neural signals, and if verified, the wearable system can (reliably) use the information gleaned from the real-time generated neural signal data to perform a variety of tasks (e.g., improving auditory perception by helping the user improve audio processing, particularly in noisy environments; identifying which person the user is trying to pay attention to in a noisy environment and amplifying the corresponding audio while attenuating other sounds).
[0034] In-ear EEG is a method of recording brain activity. The IED of a wearable system includes sensors (e.g., in-ear electrodes) located in the ear canal that pick up activity from a large number of neurons that discharge synchronously in response to a stimulus (e.g., multiple audible clicks). EEG signal data provides an image of electrical activity in the brain, represented as waves with frequency, amplitude, and shape that vary over time. This can be used to measure brain activity during events, such as completing a task or presenting a stimulus such as a speech, or to measure spontaneous brain activity that occurs in the absence of a specific event. However, current EEG systems do not have a fast "online" or "portable, real-time" method for determining that a user is recording or providing an intention neural signal to a wearable system. In fact, most of the signal processing and preprocessing in current systems is performed offline after the electrical signal data from the participant has been collected. Therefore, the signal quality cannot be verified in real time. In order to overcome the above problems, the present disclosure provides a wearable system: the wearable system is capable of acquiring and generating EEG signal data, calibrating the EEG signal data in real time by analyzing the EEG signal data, and performing actions based on the analysis.
[0035] The wearable system may include one or more controllers. Each controller may include one or more processors. The one or more controllers of the wearable system may be configured individually or in combination to perform functions that implement real-time in-ear EEG signal verification operations according to the present disclosure. The one or more controllers may include controllers in one or two of the multiple IEDs, and controllers in devices separate from the one or two IEDs (e.g., head-mounted devices, smart phones, portable electronic devices, and remote devices connected to the IEDs via a network, etc.). As described herein, a controller may refer to one or more of the controllers of the wearable system, whether the controller is on the IED or on a device separate from the IED.
[0036] The EEG signal calibration operation may be part of a startup sequence of the wearable system, or may be performed in response to a user operation. To perform the EEG signal calibration operation, the controller may control a speaker of the IED to present a calibration audio signal to the user. The calibration audio signal may be embedded with a predetermined audible feature (e.g., a music clip or other audio content synthesized or embedded with multiple clicks or chirps). When presenting the audio signal, the controller may be configured to control in-ear electrodes in one or more IEDs to collect electrical signals corresponding to the user's brain activity. The brain activity may include a time-locked neural response of the user in response to a predetermined audible feature embedded in the calibration audio signal. The controller may then use the collected electrical signals to generate an EEG signal over time.
[0037] The controller can analyze the EEG signal and perform actions based on the analysis. For example, the controller can determine (for one or two IEDs) whether the EEG signal data satisfies a predetermined amplitude suppression condition and a predetermined time condition after the predetermined audible feature begins. If so, the controller can determine that the (one or two) IEDs are well located in the ear canal (e.g., the in-ear electrodes are in good contact with the inner surface of the ear canal) and that the EEG signal data is true neural data that accurately represents the user's brain activity in response to the stimulus. And in this case, the controller can control (e.g., via a speaker or display device of the wearable system) to notify the user accordingly.
[0038] On the other hand, if the EEG signal data does not satisfy the predetermined amplitude suppression condition or the predetermined time condition after the predetermined audible feature begins, the controller can control the performance of a pitch discrimination operation to determine whether the weaker response or delayed response is due to hearing loss, or due to the IED not being well located in the ear canal, or due to an in-ear electrode failure. The pitch discrimination operation can disambiguate the case of hearing loss from the case where (one or both) IEDs are not well located in the ear canal (e.g., the in-ear electrodes are not in good contact with the inner surface of the ear canal), and after disambiguation, the controller can control (e.g., via a speaker or display device of the wearable system) to notify the user accordingly.
[0039] Therefore, before starting to use the EEG signal data for other applications (e.g., improving auditory perception), the wearable system can ensure that the generated EEG signal data is real neural data that accurately represents the user's brain activity in response to stimulation. The wearable system can also notify the user to perform various actions (e.g., adjust the position or fit of the IED in the ear canal, mark the hearing loss for further clinical evaluation, and replace the IED or check the electrode for failure, etc.) based on the analysis of the EEG signal data. Therefore, the calibration system and method provide a real-time error detection mechanism that automatically notifies the user whether the in-ear electrode of the IED is in good contact with the inner surface of the user's ear canal, and if not, whether it is necessary to readjust or reposition the electrode to make the electrode fit better with the user's tissue, or take other steps.
[0040] The various embodiments of the present disclosure may include an artificial reality system or be implemented in combination with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to the user. Artificial reality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with collected (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used to create content in artificial reality and / or used in artificial reality in other ways. Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including wearable devices (e.g., head-mounted devices) connected to a host computer system, standalone wearable devices (e.g., head-mounted devices), mobile devices or computing systems, or any other hardware platform capable of providing artificial reality content to one or more viewers.
[0041] Figure 1 1 is a block diagram of an EEG calibration system 100 according to one or more embodiments. The EEG calibration system 100 may include an in-ear device (IED) 130, a calibration device 150, and a network 170. The IED 130 is installed in the ear canal 118 of the user, near the tympanic membrane 120, and collects various types of data from the ear canal 118. The calibration device 150 includes a controller 160. The calibration device 150 receives data (e.g., EEG signal data, electrical signal data, sensor data) from the IED 130 via the network 170, analyzes the received data, and performs actions based on the analysis. Some embodiments of the IED 130 and the calibration device 150 have components that are different from those described herein. Similarly, in some cases, the functions may be distributed between the components in a manner different from that described herein. For example, some or all of the processing used by the calibration device 150 for real-time in-ear EEG signal verification may be performed by the IED 130. As another example, one or more steps of the process described herein for real-time in-ear EEG signal verification by IED 130 may be performed by calibration device 150. Figure 1The system 100 is shown to include one IED 130, but another embodiment of the system 100 may include two IEDs 130, one for each ear of the user. Figure 1 As shown, IED 130 may include audio transducer 102, in-ear electrodes 104, acoustic sensor 108, motion sensor 110, controller 112, battery 114, communication interface 116, and acoustic sensor 124. These components of IED 130 may be mounted to a circuit board (not shown) that connects the components to each other.
[0042] The audio transducer 102 is a speaker that generates sound from audio data and outputs the sound into the ear canal 118. The audio transducer 102 (e.g., a notification unit) can be used to present an audio content signal to a user. For example, the audio transducer 102 can be used to present a calibration audio signal embedded with a predetermined audible feature to the user. The calibration audio signal can be a segment of an entertainment signal (e.g., speech, music, audio, and a start or start sequence sound, etc.), and the predetermined audible feature can be a stimulus (e.g., one or more clicks, one or more chirps, and a real tone, etc.), which is known to produce a predetermined neural response in the user as an auditory evoked potential extracted from the ongoing electrical activity in the user's brain and recorded via the in-ear electrode 104. For example, the calibration audio signal is a music segment played when the IED is started / powered on, and the music segment has been processed / synthesized to embed the click sound (e.g., "click" music) by tuning the phase response so that the musicality of the segment is preserved. The audio transducer 102 can also be used to present other types of audio content to the user. For example, the user may be notified of actions determined based on the calibration operation (e.g., notifying the user that the in-ear electrode in the ear canal is in good contact, notifying the user to adjust the IED in the ear canal, notifying the user to check for hearing loss, etc.) by presenting appropriate audio content to the user via the audio transducer 102. In addition, in some embodiments, the audio transducer 102 relays sounds from a local area detected by the acoustic sensor 124, so that the IED 130 provides a hear-through function even when the ear canal 118 is blocked.
[0043] The in-ear electrodes 104 collect electrical charges generated by activity in brain cells in response to stimulation (e.g., presenting a calibrated audio signal having predetermined audible characteristics to the user through the user's ear canal 118). The electrical signals collected by the in-ear electrodes 104 can be used to generate EEG signal data that defines a waveform over time that represents electrical activity occurring within the user's brain after the stimulation (e.g., multiple audible clicks embedded in music) begins. In some embodiments, the in-ear electrodes 104 can be part of a set of in-ear electrodes that can be used to generate different types of electrograms (e.g., electroencephalogram (EEG), electrocorticography (ECoG or iEEG), electrooculogram (EOG), electroretinography (ERG), electrocardiogram (ECG)) of the brain, eyes, and heart, etc. Regardless of the type of electrogram, ensuring that the electrodes are in good contact with the user's anatomical structure is necessary to establish the validity of the data collected by the electrodes. Thus, the in-ear electrodes 104 may enable the wearable system to establish the validity of electrical signals that are indicative not only of brain activity, but also of activity in other parts of the user's anatomy (eg, eyes, heart, etc.).
[0044] The in-ear electrode 104 is located on the IED 130 so that the in-ear electrode 104 contacts the inner surface of the user's ear canal 118 when the user wears the IED 130. In some embodiments, the in-ear electrode 104 is a dry electrode that can directly contact the user's anatomical structure. The dry electrode does not require gel or some other type of medium or layer between the in-ear electrode 104 and the tissue. The in-ear electrode 104 may include a hard material electrode (e.g., including gold-plated brass, iridium oxide, etc.) or a soft material electrode and / or a stretchable material electrode (e.g., including conductive fabrics, conductive polymers, carbon allotropes such as graphene or carbon nanotubes, or poly(3,4-ethylenedioxythiophene) polystyrene sulfonate (PEDOT:PSS)).
[0045] like Figure 1As shown, the IED 130 may include two in-ear electrodes 104 to generate EEG signal data using the principle of differential amplification by recording the voltage difference between different points, which compares one active detection electrode site to another adjacent or distant reference electrode. Other embodiments of the IED 130 may include additional or fewer in-ear electrodes 104 to collect electrical signals, as long as one or more IEDs 130 can generate EEG signal data based on differential amplification. In some embodiments, multiple in-ear electrodes 104 may be located on the IED 130 to contact different locations of the inner surface of the user's ear or ear canal 118. In some embodiments, multiple electrodes can be used to contact a larger surface area within the user's ear compared to a single in-ear electrode, thereby potentially improving signal quality. As shown in FIG. Figure 1 As shown, the IED 130 may include additional components (eg, acoustic sensor 108 and motion sensor 110) for collecting other biometric data of the user (eg, EOG, ECG).
[0046] like Figure 1As shown, the IED 130 includes a controller 112 that performs processing to facilitate the collection of sensor data. For example, the controller 112 can control the in-ear electrodes 104 to receive the electrical signals collected by the in-ear electrodes 104. In some embodiments, the controller 112 may include a differential amplifier for amplifying the difference between the voltage signals detected at each in-ear electrode 104. The controller 112 may also include an analog to digital converter (ADC) that converts the electrical signals from the in-ear electrodes 104 into EEG signal data representing brain activity over time in response to stimulation. The ADC may also convert sensor data from other sensors (e.g., acoustic sensor 108 and / or motion sensor 110) into digital data representing waveforms. The controller 112 may be configured to perform additional processing, such as playing audio content, recording audio content, collecting sensor data, and performing predetermined processing on sensor data. The controller 112 may also include a digital to analog converter (DAC) that converts digital audio data into analog audio data for rendering by the audio transducer 102. For example, the controller 112 may provide a calibration audio signal (e.g., a clip of "click" music) or other audio content to the audio transducer 102 for rendering to the user. In some embodiments, the controller 112 is also configured to determine when to perform an EEG signal calibration operation. For example, upon determining that the IED 130 has been powered on, the controller 112 may control the audio transducer 102 to present the calibration audio signal to the user for at least a threshold amount of time (e.g., about 30 seconds). As another example, upon determining (e.g., by the motion sensor 110) that the IED 130 has been placed in the ear canal 118 of the user, the controller may control the audio transducer 102 to present the calibration audio signal. In some embodiments, when presenting the calibration audio signal to the user, the controller may also control the in-ear electrodes 104 to collect electrical signals corresponding to the user's brain activity in response to the presented calibration audio signal. One or more of the features of the controller 112 may be performed by the controller 160 of the calibration device 150. Figure 3 1 and 12 to describe additional features of the controller 112 according to some embodiments.
[0047] The battery 114 provides power to other components of the IED 130. The battery 114 allows the IED 130 to operate as a mobile device. The battery 114 may be charged via wire or wirelessly.
[0048] The communication interface 116 facilitates connecting the IED 130 to other devices, such as the calibration device 150, via the network 170 (e.g., wirelessly). For example, the communication interface 116 can transmit data (e.g., sensor data) generated by the IED 130 to the calibration device 150 for analysis and execution of actions based on the analysis. The IED 130 can also receive notification data, audio content data, calibration audio signal data, or other types of information from the calibration device 150 via the communication interface 116. In some embodiments, the communication interface 116 includes an antenna and a transceiver.
[0049] The calibration device 150 may be configured to receive signals from the in-ear electrodes 104 and / or other sensors (e.g., from the IED 130 and / or from the head-mounted device ( Figure 2A and Figure 2B )) collected data and generates EEG signal data and / or analyzes EEG signal data, which is used for real-time in-ear EEG signal verification. In one embodiment, the calibration device 150 is a head-mounted device or a head-mounted display (HMD), as shown below in combination Figure 2A and Figure 2B Alternatively, the calibration device 150 may be a device having computer functionality, such as a desktop computer, a laptop computer, a personal digital assistant (PDA), a mobile phone, a smart phone, a tablet computer, an Internet of Things (IoT) device, a virtual conferencing device, a cuff, or another suitable device.
[0050] The controller 160 may include various components that provide functionality for in-ear EEG signal verification. These components may include, for example, one or more processors, a data repository 155, a calibration module 165, a signal processing module 166, and a pitch discrimination module 180. Some embodiments of the controller 160 have components that are different from those described herein. Similarly, in some cases, the functions may be distributed among the components in a manner different from that described herein.
[0051] The data repository 155 stores data used by the calibration device 150 (e.g., EEG signal data, calibration audio signal data, and program instruction data corresponding to various modules of the controller 160, etc.). The data repository 155 may also store data used by the IED 130. The data repository 155 (e.g., a non-transitory computer-readable storage medium) and one or more processors operate in conjunction to perform various functions attributed to the calibration device 150 as described herein. For example, the data repository 155 may store one or more modules or applications embodied as instructions executable by one or more processors of the controller 160. These instructions, when executed by the controller 160, cause the controller 160 to perform the functions of the various modules or applications attributed to the controller 160.
[0052] The calibration module 165 can perform real-time in-ear EEG signal verification by causing a calibration audio signal to be presented to the user. In order to synthesize the calibration audio signal with an audible stimulus (e.g., a click), the phase response of the sound can be aligned so that a burst (e.g., a click) of sufficient intensity to induce a neural response is incorporated into the audio signal while the overall amplitude of the audio signal remains unchanged. The calibration audio signal (synthesized with predetermined audible characteristics) can be pre-generated and stored in the data repository 155 for presentation to the user, or it can be generated "on the fly" by the controller 160. The calibration module 165 can generate EEG signal data based on the electrical signals collected by the in-ear electrodes 104 and the electrical signals received from the IED 130. Alternatively, the EEG signal data can be generated by the controller 112 of the IED 130, and the data can be received by the calibration module 165 for storage in the data repository 155 and further analysis.
[0053] The signal processing module 166 can be configured to pre-process the generated or received EEG signals. For example, the EEG data can undergo online pre-processing to "clean the data" and remove any noise to separate the real neural data. In some embodiments, where the calibration module 165 generates EEG signal data, the signal processing module 166 can pre-process the electrical signals collected by the in-ear electrodes 104 and the electrical signals from the IED 130 received by the calibration module 165, and generate EEG signal data from the pre-processed electrical signals to reduce noise and / or separate noise from the real neural data.
[0054] The calibration module 165 may then analyze the cleansed EEG signal data to determine the quality of the EEG signal data, and perform actions based on the quality determination analysis. In some embodiments, to determine the quality of the signal data, the calibration module 165 may include tools for generating brain activity signatures from the cleansed EEG signal data, and identifying robust amplitude and timing information of the data over time. The calibration module 165 may then compare the generated brain activity signatures showing amplitude and timing information with predetermined threshold brain activity signatures to determine whether the characteristics of the EEG signal data, such as amplitude and timing, indicate that the user's neural response to the audible stimulus is time-locked as expected. For example, the calibration module 165 may be configured to determine signal quality by determining whether the (averaged) EEG signal data exhibits a predetermined amplitude suppression at approximately 50 ms after the onset of a click stimulus embedded in the music. Therefore, if the electrical signal collected in one IED 130 is weaker than the electrical signal collected in another IED 130 and / or weaker than a threshold value, it may indicate that one or both of the IEDs 130 are not well installed in the ear canal, and the controller 160 may prompt the user accordingly (e.g., via a speaker or display device of the system). As another example, if the electrical signal collected in one IED 130 is delayed and / or delayed by more than a threshold value than the electrical signal collected in another IED 130, it may indicate that one or both of the IEDs 130 are not well installed in the ear canal, and the controller 160 may prompt the user accordingly (e.g., via a speaker or display device of the system). Therefore, the calibration module 165 may use the collected electrical signal to verify the operation and positioning of the IED 130 in the ear canal.
[0055] The calibration module 165 may also be configured to perform different actions based on the quality determination or analysis. In some embodiments, the calibration module 165 may include an automatic classifier (an artificial intelligence classifier or a machine learning-based classifier) for determining whether the generated EEG signal data indicates that the user's time-locked neural response to the stimulus is as expected. For example, if the EEG signal does not show an amplitude suppression weaker than the threshold amplitude suppression after the (click) stimulus embedded in the music begins, and does not show a delayed response longer than the threshold response delay, the calibration module 165 can determine that the in-ear electrode 104 is in good contact with the inner surface of the user's ear canal 118. In this case, the calibration module 165 can also determine that the generated EEG signal data is true neural data that accurately represents the user's brain activity in response to the stimulus (e.g., the impedance and / or signal-to-noise ratio of the electrical signal from the in-ear electrode meets the minimum threshold level). The calibration module 165 can also be configured to present a notification of this result to the user (e.g., instructing the speaker 102 or the display device to notify the user that the IED is well fitted in the ear canal 118).
[0056] On the other hand, if the EEG signal shows an amplitude suppression weaker than the threshold amplitude suppression, shows a delayed response longer than the threshold response delay, or both after the click stimulus embedded in the music starts, the calibration module 165 can cause the pitch discrimination module 180 to perform a pitch discrimination operation. In order to eliminate the ambiguity between the user's hearing loss and the electrode function or the poor contact between the in-ear electrode 104 and the inner surface of the ear canal 118, the controller 160 can instruct the pitch discrimination module 180 to perform a pitch discrimination task to test the user's hearing ability. For example, the pitch discrimination module 180 can control the output of a predetermined audio signal from the speaker of the IED 130, while changing the pitch of the audio (e.g., changing the pitch to a frequency higher than 4000 Hz), and prompt the user to make a selection (e.g., pitch increase or decrease) to determine whether the user can distinguish audio of different pitches. Thus, since a user with hearing loss may not be able to properly handle this task, the calibration device 150 can accurately disambiguate the case of hearing loss from the case where the IED 130 is not properly seated in the ear canal 118 (or the electrode is faulty).
[0057] If the user is able to successfully complete the pitch discrimination task performed by the pitch discrimination module 180, the calibration module 165 can determine that the delayed response, the weak response, or both are due to the IED 130 not being well installed in the ear canal 118 (or an electrode failure). The calibration module 165 can also be configured to present a notification of this result to the user (e.g., instructing the speaker 102 or a display device to notify the user to readjust or reposition the IED 130 in the ear canal 118 to achieve a better fit and contact between the in-ear electrode 104 and the inner surface of the ear canal 118, instructing the speaker 102 to notify the user to replace the IED 130 due to an electrode failure, or both).
[0058] If the user is unable to successfully complete the pitch discrimination task performed by the pitch discrimination module 180, the calibration module 165 can determine that the weak response / delayed response is due to hearing impairment. The calibration module 165 can also be configured to present a notification of this result to the user (e.g., instruct the speaker 102 or display device to notify the user of the impairment). And in this case, the calibration module 165 can also be configured to operate the signal processing module 166 to correct (e.g., amplify) the gain of a predetermined frequency by performing signal processing, thereby adjusting the audio from the speaker 102 presented to the user (with hearing loss) to compensate for the user's hearing loss. For example, audio content can be selectively presented to the user, and signals from in-ear electrodes can be monitored to determine an audio profile for the user (e.g., measuring the degree to which the user hears the sound based on frequency). Then, the audio profile for the user can be used to form audio content for the user.
[0059] Some or all components of the calibration device 150 may be located in the IED 130. That is, some or all functions of the calibration device 150 may be performed by the IED 130. That is, the controller 160 may be an embodiment of the controller 112.
[0060] In some embodiments, the calibration device 150 is a server connected to the IED 130 via a network 170 including the Internet. The network 170 may include any combination of a local area network and / or a wide area network using a wired communication system and / or a wireless communication system. In one embodiment, the network 170 uses standard communication technologies and / or protocols. For example, the network 170 includes communication links using a variety of technologies, such as Ethernet, 802.11 (WiFi), worldwide interoperability for microwave access (Wi MAX), 3rd generation mobile communication technology (3G), 4th generation mobile communication technology (4G), 5th generation mobile communication technology (5G), code division multiple access (CDMA), digital subscriber line (DSL), Bluetooth (BLUETOOTH), Near Field Communication (NFC), Universal Serial Bus (USB), or any combination of multiple protocols. In some embodiments, all or some communication links of the network 170 may be encrypted using any suitable technology or technologies.
[0061] Figure 2A is a stereoscopic diagram of a head mounted device 200 implemented as an eyewear device according to one or more embodiments. The head mounted device 200 is an example of a calibration device 150. In some embodiments, the eyewear device is a near eye display (NED). Generally speaking, the head mounted device 200 can be worn on the face of a user so that content (e.g., media content) is presented using a display component and / or an audio system. However, the head mounted device 200 can also be used so that media content is presented to the user in a different manner. Examples of media content presented by the head mounted device 200 include one or more images, videos, audio, or some combination thereof. The head mounted device 200 includes a frame and may include a display component (the display component includes one or more display elements 220 (e.g., a notification unit)), a depth camera assembly (DCA), an audio system, and other components such as a position sensor 290. Although Figure 2A The components of the head mounted device 200 are shown as being located at example locations on the head mounted device 200, but the components may be located at other locations on the head mounted device 200, on a peripheral device paired with the head mounted device 200 (e.g., on the IED 130), or some combination thereof. Figure 2A More components or fewer components than those shown may be used.
[0062] The frame 210 holds the other components of the head mounted device 200. The frame 210 includes: a front portion that holds one or more display elements 220, and end pieces (e.g., temples) that attach to the user's head. The front portion of the frame 210 spans the top of the user's nose. The length of the end piece can be adjustable (e.g., adjustable temple length) to fit different users. The end piece can also include a portion that bends behind the user's ear (e.g., temple end, earpiece).
[0063] like Figure 2A As shown, frame 210 may include one or more electrodes 245 . Figure 2A The embodiment shown in shows that two electrodes 245 are located in the nose pads of the frame 210 and two electrodes 245 are located at the ends of the temples. However, this is not intended to be limiting. Other embodiments of the head-mounted device 200 may have fewer or more electrodes 245, which may be arranged at positions other than the nose pads and temple ends of the frame 210, or arranged at the nose pads and temple ends of the frame 210 and other external locations other than the nose pads and temples. Each electrode 245 may be installed to be configured to contact the user's anatomical structure (e.g., bridge of the nose, behind the ear) when the user wears the head-mounted device 200. Similar to the in-ear electrode 104, each electrode 245 may be configured to collect the charge generated by the activity (e.g., neural response) in the brain cells in response to the stimulation. In some embodiments, the head-mounted device 200 may be configured to generate EEG signal data based on the electrical signals collected by one or more electrodes in each electrode 245 and based on the electrical signals collected by the in-ear electrode 104. Electrodes 245 may be located anywhere on frame 210 as long as they contact the user's tissue to enable acquisition of electrical signals representing the user's brain activity time-locked in response to the audible stimulus. Other aspects of electrodes 245 may be similar to those of in-ear electrodes 104. In some embodiments, electrodes 245 may replace in-ear electrodes 104 to acquire electrical signals for generating EEG signals.
[0064] One or more display elements 220 provide light to a user wearing the head mounted device 200. As shown, the head mounted device 200 includes a display element 220 for each eye of the user. In some embodiments, the display element 220 generates image light, which is provided to the eyebox of the head mounted device 200. The eyebox is the position in space occupied by the eyes of the user when the head mounted device 200 is worn. For example, the display element 220 can be a waveguide display. The waveguide display includes a light source (e.g., a two-dimensional source, one or more line sources, one or more point sources, etc.) and one or more waveguides. Light from the light source is coupled into one or more waveguides, which output light in a manner such that there is pupil replication in the eyebox of the head mounted device 200. The coupling of light and / or the coupling out of light from one or more waveguides can be accomplished using one or more diffraction gratings. In some embodiments, the waveguide display includes a scanning element (e.g., a waveguide, a reflector, etc.) that scans the light from the light source when it is coupled into the one or more waveguides. Note that in some embodiments, one or two of the display elements 220 are opaque and do not transmit light from a local area around the head-mounted device 200. The local area is the area around the head-mounted device 200. For example, the local area may be a room where a user wearing the head-mounted device 200 is located, or the user wearing the head-mounted device 200 may be outdoors and the local area is an outdoor area. In this context, the head-mounted device 200 generates VR content. Alternatively, in some embodiments, one or two of the display elements 220 are at least partially transparent so that light from the local area can be combined with light from the one or more display elements to generate AR content and / or MR content.
[0065] In some embodiments, the display element 220 does not generate image light, but the display element is a lens that transmits light from a local area to the eye zone. For example, one or two of the display elements 220 can be an uncorrected (non-prescription) lens or a prescription lens (e.g., a single-vision lens, a bifocal and trifocal lens, or a progressive lens) that helps correct defects in the user's vision. In some embodiments, the display element 220 can be polarized and / or colored to protect the user's eyes from sun damage. In some embodiments, the display element 220 may include an additional optical block (not shown). The optical block may include one or more optical elements (e.g., a lens, a Fresnel lens, etc.) that guide light from the display element 220 to the eye zone. The optical block can, for example, correct aberrations in part or all of the image content, magnify part or all of the image, or some combination thereof.
[0066] The DCA determines depth information of a portion of a local area around the head mounted device 200. The DCA includes one or more imaging devices 230 and a DCA controller ( Figure 2A 2 (not shown), and may also include an illuminator 240. In some embodiments, the illuminator 240 illuminates a portion of the local area with light. The light may be, for example, structured light in infrared (IR) (e.g., dot pattern structured light, strip structured light, etc.), IR flash for time of flight, etc. In some embodiments, one or more imaging devices 230 capture an image of a portion of the local area that includes light from the illuminator 240. As shown, Figure 2A A single illuminator 240 and two imaging devices 230 are shown. In an alternative embodiment, there is no illuminator 240 and there are at least two imaging devices 230. The DCA controller uses the acquired images and one or more depth determination techniques to calculate depth information for the portion of the local area. The depth determination techniques may be, for example, direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (using texture added to the scene by light from the illuminator 240), some other technique for determining the depth of the scene, or some combination thereof.
[0067] The DCA may include an eye tracking unit that determines eye tracking information. The eye tracking information may include information about the position and orientation of one or both eyes (within their respective eye-adaptive zones). The eye tracking unit may include one or more cameras. The eye tracking unit estimates the angular orientation of one or both eyes based on images of the one or both eyes captured by one or more cameras. In some embodiments, the eye tracking unit may also include one or more illuminators that illuminate one or both eyes with an illumination pattern (e.g., structured light, flicker, etc.). The eye tracking unit may use the illumination pattern in the captured image to determine the eye tracking information. The head mounted device 200 may prompt the user to opt-in to allow operation of the eye tracking unit. For example, by opting-in, the head mounted device 200 may detect and store images of the user's eyes or the user's eye tracking information.
[0068] In some embodiments, although Figure 2ANot shown, the head mounted device 200 may include one or more electrooculogram (EOG) electrodes positioned proximate to the user's eyes and configured to measure electrical signals representing a corneal-retinal resting potential present between the front and back of one or both eyes of the user to generate EOG signal data. The EOG signal data is temporally correlated with the gaze direction of the user's eyes. The eye tracking unit may also be configured to determine eye tracking information based on the EOG signal data generated using the EOG electrodes. In some embodiments, the eye tracking information and the EEG signal data corresponding to the in-ear electrodes 104 and / or electrodes 245 may be used together, for example, to determine whether the user is paying attention to a particular sound source. Based on this determination, the audio system of the head mounted device 200 may selectively highlight sounds from the identified sound source relative to other sounds in the local area (e.g., beamforming).
[0069] The audio system provides audio content. The audio system includes a transducer array, a sensor array, and an audio controller 250. However, in other embodiments, the audio system may include different components and / or additional components. Similarly, in some cases, the functions described with reference to the components in the audio system may be distributed among these components in a manner different from that described herein. For example, some or all of the functions of the controller may be performed by a remote server.
[0070] The transducer array presents sound to the user. The transducer array includes multiple transducers. The transducer can be a speaker 260 or a tissue transducer 270 (e.g., a bone conduction transducer or a cartilage conduction transducer). Although the speaker 260 is shown as being outside the frame 210, the speaker 260 can be enclosed in the frame 210. In some embodiments, the head mounted device 200 includes a speaker array instead of a separate speaker for each ear, the speaker array including multiple speakers integrated into the frame 210 to improve the directionality of the presented audio content. The tissue transducer 270 is coupled to the user's head and directly vibrates the user's tissue (e.g., bone or cartilage) to produce sound. The number and / or position of the transducers can vary depending on the user's head. Figure 2A In some embodiments, the transducer (eg, audio transducer 102 ) is located in IED 130 .
[0071] The sensor array detects sounds in a local area of the head mounted device 200. The sensor array includes a plurality of acoustic sensors 280. The acoustic sensor 280 collects sounds emitted from one or more sound sources in a local area (e.g., a room). Each acoustic sensor is configured to detect sounds and convert the detected sounds into an electronic format (analog or digital). The acoustic sensor 280 may be an acoustic wave sensor, a microphone, a sound transducer, or a similar sensor suitable for detecting sounds. In some embodiments, the acoustic sensor 124 and the acoustic sensor 108 are substantially the same as the acoustic sensor 280, except that the acoustic sensors 108, 124 are integrated into the IED and the acoustic sensor 280 is integrated into the head mounted device 200.
[0072] In some embodiments, one or more acoustic sensors may be placed in the ear canal of each ear (e.g., one or more acoustic sensors may be placed in the IED 130 in the ear canal of each ear, acting as a binaural microphone). In some embodiments, the acoustic sensors 280 may be placed on an exterior surface of the head mounted device 200, on an interior surface of the head mounted device 200, separate from the head mounted device 200 (e.g., part of some other device), or some combination thereof. The number and / or location of the acoustic sensors 280 may vary depending on the device. Figure 2A . For example, the number of acoustic detection locations can be increased to increase the amount of audio information collected and the sensitivity and / or accuracy of the information. The acoustic detection locations can be oriented so that the microphones can detect sounds in a wide range of directions around the user wearing the head mounted device 200.
[0073] The audio controller 250 processes information from the sensor array describing the sound detected by the sensor array. The audio controller 250 may include a processor and a computer-readable storage medium. The audio controller 250 may be configured to generate a direction of arrival (DOA) estimate, generate an acoustic transfer function (e.g., an array transfer function and / or a head-related transfer function), track the location of a sound source, form a beam in the direction of a sound source, classify the sound source, generate a sound filter for the speaker 260, or some combination thereof. In some embodiments, the audio controller 250 may include some or all of the functions provided by the controller 160 of the calibration device 150 and / or by the controller 112 of the IED 130. Therefore, the audio controller 250 may be configured to perform a real-time in-ear EEG signal verification operation. In some embodiments, some or all of the functions of the audio controller 250 may be provided by the controller 112 of the IED 130.
[0074] The position sensor 290 generates one or more measurement signals in response to the movement of the head mounted device 200. The position sensor 290 may be located on a portion of the frame 210 of the head mounted device 200. The position sensor 290 may include an inertial measurement unit (IMU). Examples of the position sensor 290 include: one or more accelerometers; one or more gyroscopes; one or more magnetometers; other suitable types of sensors that detect movement; a type of sensor used for error correction of the IMU; or some combination thereof. The position sensor 290 may be located outside the IMU, inside the IMU, or some combination thereof.
[0075] In some embodiments, the head mounted device 200 can provide simultaneous localization and mapping (SLAM) for the position of the head mounted device 200 and update the model of the local area. For example, the head mounted device 200 may include a passive camera assembly (PCA) that generates color image data. The PCA may include one or more RGB cameras that capture images of part or all of the local area. In some embodiments, some or all of the imaging devices 230 of the DCA may also be used as PCA. The images captured by the PCA and the depth information determined by the DCA can be used to determine parameters of the local area, generate a model of the local area, update the model of the local area, or some combination thereof. In addition, the position sensor 290 tracks the positioning (e.g., position and posture) of the head mounted device 200 in the room.
[0076] Figure 2B 2 is a perspective view of a head mounted device 205 implemented as an HMD according to one or more embodiments. In embodiments describing an AR system and / or an MR system, multiple portions of the front face of the HMD are at least partially transparent in the visible wavelength band (approximately 380 nanometers (nm) to 750 nm), and multiple portions of the HMD located between the front face of the HMD and the user's eyes are at least partially transparent (e.g., partially transparent electronic displays). The HMD includes a front rigid body 215 and a band 275. The head mounted device 205 includes many of the same components as those described above. Figure 2A The components described are the same components, but these components are modified to be integrated with the HMD form factor. For example, the HMD includes a display component, a DCA, an audio system, EEG electrodes, EOG electrodes, and a position sensor 290. Figure 2BShown are an illuminator 240, a plurality of speakers 260, a plurality of imaging devices 230, a plurality of acoustic sensors 280, and a position sensor 290. The speakers 260 may be located in a variety of locations, such as coupled to the band 275 (as shown), coupled to the front rigid body 215, or may be configured to be inserted into the ear canal of a user.
[0077] Figure 3 is a block diagram of an audio system 300 according to one or more embodiments. The audio system 300 may include Figure 1 All or part of the functions of the controller 160 of the calibration device 150, and / or Figure 1 All or part of the functions of the controller 112 of the IED 130. In addition, Figure 2A or Figure 2B The audio system in may be an embodiment of the audio system 300. The audio system 300 generates one or more acoustic transfer functions for a user. The audio system 300 may then use the one or more acoustic transfer functions to generate audio content for the user. Figure 3 In an embodiment of the present invention, the audio system 300 includes a transducer array 310, a sensor array 320, and an audio controller 330. Some embodiments of the audio system 300 have different components than those described herein. Similarly, in some cases, the functions may be distributed among the components in a manner different from that described herein.
[0078] The transducer array 310 is configured to present audio content. The transducer array 310 includes a plurality of transducers. A transducer is a device that provides audio content. The transducer may be, for example, a speaker (e.g., speaker 102, speaker 260), a tissue transducer (e.g., tissue transducer 270), some other device that provides audio content, or some combination thereof. The tissue transducer may be configured to be used as a bone conduction transducer or a cartilage conduction transducer. The transducer array 310 may present audio content via air conduction (e.g., via one or more speakers), via bone conduction (via one or more bone conduction transducers), via a cartilage conduction audio system (via one or two cartilage conduction transducers), or some combination thereof. In some embodiments, the transducer array 310 may include one or more transducers to cover different parts of a frequency range. For example, a piezoelectric transducer may be used to cover a first part of a frequency range, and a moving coil transducer may be used to cover a second part of a frequency range.
[0079] The bone conduction transducer generates sound pressure waves by vibrating the bones / tissue of the user's head. The bone conduction transducer can be coupled to a portion of the head mounted device and can be configured to be located behind the auricle coupled to a portion of the user's skull. The bone conduction transducer receives vibration instructions from the audio controller 330 and vibrates a portion of the user's skull based on the received instructions. The vibrations from the bone conduction transducer generate tissue-propagated sound pressure waves that bypass the eardrum and propagate toward the user's cochlea.
[0080] The bone conduction transducer generates sound pressure waves by vibrating one or more parts of the ear cartilage of the user's ear. The bone conduction transducer can be coupled to a portion of the head-mounted device and can be configured to be coupled to one or more parts of the ear cartilage of the ear. For example, the bone conduction transducer can be coupled to the back of the auricle of the user's ear. The bone conduction transducer can be located anywhere along the ear cartilage around the outer ear (e.g., the auricle, the tragus, some other part of the ear cartilage, or some combination thereof). Vibrating one or more parts of the ear cartilage can generate: air-transmitted sound pressure waves outside the ear canal; tissue-transmitted sound pressure waves that cause certain parts of the ear canal to vibrate, thereby generating air-transmitted sound pressure waves in the ear canal; or some combination thereof. The generated air-transmitted sound pressure waves propagate along the ear canal toward the eardrum.
[0081] The transducer array 310 generates audio content according to instructions from the audio controller 330. In some embodiments, the audio content is spatialized. The spatialized audio content is audio content that sounds like it originates from a specific direction and / or target area (e.g., objects and / or virtual objects in a local area). For example, the spatialized audio content can make it sound like it originates from a virtual singer located at the other end of the user's room of the audio system 300. The transducer array 310 can be coupled to a wearable device (e.g., IED 130, head mounted device 200, or head mounted device 205). In an alternative embodiment, the transducer array 310 can be a plurality of speakers that are separate from the wearable device (e.g., coupled to an external console).
[0082] The sensor array 320 detects sounds in a local area around the sensor array 320. The sensor array 320 may include a plurality of acoustic sensors (e.g., sensors 108, 124, and / or 280), each of which detects changes in air pressure of sound waves and converts the detected sounds into an electronic format (analog or digital). The plurality of acoustic sensors may be located on a head mounted device (e.g., head mounted device 200 and / or head mounted device 205), on a user (e.g., an IED 130 in the user's ear canal), on a neckband, or some combination thereof. The acoustic sensor may be, for example, a microphone, a vibration sensor, an accelerometer, or any combination thereof. In some embodiments, the sensor array 320 is configured to monitor the audio content generated by the transducer array 310 using at least some of the plurality of acoustic sensors. Increasing the number of sensors may improve the accuracy of the information (e.g., directionality) describing the sound field generated by the transducer array 310 and / or the sound from the local area.
[0083] The audio controller 330 controls the operation of the audio system 300. Figure 3 In the embodiment of the present invention, the audio controller 330 includes a data repository 335, a DOA estimation module 340, a transfer function module 350, a tracking module 360, a beamforming module 370, and a sound filter module 380. In the embodiment in which the audio controller 330 includes the functionality of the calibration device 150, the audio controller 330 may also include a calibration module 165, a signal processing module 166, and a pitch recognition module 180, and the data repository 335 may store data corresponding to the data repository 155. Figure 1 , Figure 2A and Figure 2B A detailed description of the components and features of the audio controller 330 discussed herein is provided to avoid repetition. In some embodiments, the audio controller 330 may be located inside the head mounted device and / or the IED 130. Some embodiments of the audio controller 330 have components that are different from those described herein. Similarly, the functions may be distributed among the components in a manner different from that described herein. For example, some functions of the controller may be performed outside the head mounted device. The user may choose to opt-in to allow the audio controller 330 to transmit data collected by the head mounted device to a system outside the head mounted device, and the user may choose a privacy setting for controlling access to any such data.
[0084] The data repository 335 stores data for use by the audio system 300. The data in the data repository 335 may include sounds recorded in a local area of the audio system 300, audio content, head-related transfer function (HRTF), transfer function of one or more sensors, array transfer function (ATF) of one or more acoustic sensors in each acoustic sensor, sound source location, virtual model of the local area, direction of arrival estimation results, sound filters, and other data related to the use of the audio system 300, or any combination thereof. The data in the data repository 335 may also include data stored in the data repository 155 and related to real-time in-ear EEG signal verification operations. For example, the data may include audio data of one or more calibration audio signals embedded with predetermined audible features (e.g., click sounds), data for the pitch recognition module 180 to perform pitch recognition tests, data for performing calibration operations (e.g., program instructions for generating EEG data from electrical signals collected by electrodes, program instructions for pre-processing electrical signals and / or EEG data, program instructions for analyzing EEG data (e.g., via an automatic classifier), threshold feature neural data (e.g., threshold amplitude conditions, threshold time conditions), program instructions for performing different actions based on the analysis, and notification data for notifying the user of the determined action).
[0085] The DOA estimation module 340 is configured to locate the sound sources in the local area based in part on the information from the sensor array 320. Localization is the process of determining the location of the sound source relative to the user of the audio system 300. The DOA estimation module 340 performs a DOA analysis to locate one or more sound sources in the local area. The DOA analysis may include analyzing the intensity, spectrum, and / or arrival time of each sound at the sensor array 320 to determine the direction from which the sound originates. In some cases, the DOA analysis may include any suitable algorithm for analyzing the surrounding acoustic environment in which the audio system 300 is located.
[0086] For example, the DOA analysis may be designed to receive input signals from the sensor array 320 (or from sensors or electrodes in the IED 130, the head mounted device 200, or the head mounted device 205) and apply digital signal processing algorithms to these input signals to estimate the direction of arrival. These algorithms may include, for example, a delayed sum algorithm in which the input signal is sampled and the final weighted and delayed versions of the sampled signals are averaged together to determine the DOA. A least mean squared (LMS) algorithm may also be implemented to create an adaptive filter. The adaptive filter may then be used to identify, for example, differences in signal strength or differences in arrival time. These differences may then be used to estimate the DOA. In another embodiment, the DOA may be determined by converting the input signal into the frequency domain and selecting specific bins in the time-frequency (TF) domain for processing. Each selected TF bin may be processed to determine whether the bin includes a portion of the audio spectrum having a direct path audio signal. Those intervals that have a portion of the direct path signal can then be analyzed to identify the angle at which the sensor array 320 receives the direct path audio signal. The determined angle can then be used to identify the DOA of the received input signal. Other algorithms not listed above can also be used alone or in combination with the above algorithms to determine the DOA.
[0087] In some embodiments, the DOA estimation module 340 can also determine the DOA relative to the absolute position of the audio system 300 (or IED 130, or head mounted device 200 or 205) within the local area. The position of the sensor array 320 can be received from an external system (e.g., some other component of the head mounted device, an artificial reality console, a map building server, a position sensor (e.g., position sensor 290), etc.). The external system can create a virtual model of the local area in which the local area and the position of the audio system 300 are drawn. The received position information can include the position and / or orientation of some or all parts of the audio system 300 (e.g., sensor array 320). The DOA estimation module 340 can update the estimated DOA based on the received position information.
[0088] The transfer function module 350 is configured to generate one or more acoustic transfer functions. In general, a transfer function is a mathematical function that gives a corresponding output value for each possible input value. The transfer function module 350 generates one or more acoustic transfer functions associated with the audio system based on the parameters of the detected sound. The acoustic transfer function can be an array transfer function (ATF), a head-related transfer function (HRTF), other types of acoustic transfer functions, or some combination thereof. The ATF characterizes how a microphone receives sound from a point in space.
[0089] ATF includes multiple transfer functions, which characterize the relationship between the sound source and the corresponding sound received by each acoustic sensor in the sensor array 320. Therefore, for a sound source, each of the acoustic sensors in the sensor array 320 has a corresponding transfer function. And this group of transfer functions is collectively referred to as ATF. Therefore, for each sound source, there is a corresponding ATF. Note that the sound source can be, for example, someone or something, a user, or one or more transducers in the transducer array 310 that generates sound in a local area. Since the physiological structure of a person (e.g., ear shape, shoulders, etc.) affects the sound when the sound travels to the human ear, the ATF relative to the specific sound source position of the sensor array 320 may vary from user to user. Therefore, each ATF of the sensor array 320 is personalized for each user of the audio system 300.
[0090] In some embodiments, the transfer function module 350 determines one or more HRTFs for a user of the audio system 300. HRTFs characterize how an ear receives sound from a point in space. Since a person's physiological structure (e.g., ear shape, shoulders, etc.) affects the sound as it travels to the person's ear, the HRTF relative to a person's specific source position is unique for each ear of the person (and thus unique for the person). In some embodiments, the transfer function module 350 can use a calibration process to determine the user's HRTF. In some embodiments, the transfer function module 350 can provide information about the user to a remote system. The user can adjust privacy settings to allow or prevent the transfer function module 350 from providing information about the user to any remote system. The remote system uses, for example, machine learning to determine a set of HRTFs customized for the user and provides the customized set of HRTFs to the audio system 300.
[0091] The tracking module 360 is configured to track the position of one or more sound sources. The tracking module 360 can compare each current DOA estimation result and compare these current DOA estimation results with the stored history of previous DOA estimation results. In some embodiments, the audio system 300 can recalculate the DOA estimation results according to a periodic schedule (e.g., once per second or once per millisecond). The tracking module can compare the current DOA estimation result with the previous DOA estimation result, and the tracking module 360 can determine that the sound source has moved in response to the change of the DOA estimation result of the sound source. In some embodiments, the tracking module 360 can detect the change of position based on visual information received from the head-mounted device or some other external source. The tracking module 360 can track the movement of one or more sound sources over time. The tracking module 360 can store the number of sound sources and the position of each sound source at each time point. The tracking module 360 can determine that the sound source has moved in response to the change of the number of sound sources or the position. The tracking module 360 can calculate the estimation result of the positioning variance. The positioning variance can be used as the confidence level for each determination of the movement change.
[0092] The beamforming module 370 is configured to process one or more ATFs to selectively highlight sounds from sound sources within a certain area while not highlighting sounds from other areas. When analyzing the sounds detected by the sensor array 320, the beamforming module 370 can combine information from different acoustic sensors to highlight sounds associated with specific areas of the local area while not highlighting sounds from outside the area. The beamforming module 370 can isolate audio signals associated with sounds from specific sound sources from other sound sources in the local area based on, for example, different DOA estimation results from the DOA estimation module 340 and the tracking module 360, eye tracking information from the eye tracking unit, (calibrated and verified) real-time EEG signal data, and EOG signal data. In some embodiments, the beamforming module 370 can isolate audio signals associated with sounds from a particular sound source based on eye tracking information generated by an eye tracking unit of the head mounted device 200 of FIG. 2 , and / or based on EEG signal data corresponding to the in-ear electrodes 104 and / or electrodes 245 and / or EOG electrodes of the head mounted device 200 . Thus, the beamforming module 370 can selectively analyze discrete sound sources in a local area. In some embodiments, the beamforming module 370 can enhance signals from a sound source. For example, the beamforming module 370 can apply sound filters that eliminate signals above certain frequencies, signals below certain frequencies, or signals between certain frequencies. Signal enhancement is used to enhance the sounds associated with a given identified sound source relative to other sounds detected by the sensor array 320 .
[0093] The sound filter module 380 determines the sound filter of the transducer array 310. In some embodiments, the sound filter spatializes the audio content so that the audio content appears to originate from the target area. The sound filter module 380 can generate the sound filter using HRTF and / or acoustic parameters. Acoustic parameters describe the acoustic characteristics of a local area. Acoustic parameters may include, for example, reverberation time, reverberation level, room impulse response, etc. In some embodiments, the sound filter module 380 calculates one or more of these acoustic parameters. In some embodiments, the sound filter module 380 requests the acoustic parameters from the map building server. The sound filter module 380 provides the sound filter to the transducer array 310 (or to the speaker of the IED 130, the head mounted device 200, or the head mounted device 205). In some embodiments, the sound filter can positively amplify or negatively amplify the sound according to the frequency.
[0094] Figure 4 is a flow diagram of a process 400 for calibrating neural signal data according to one or more embodiments. Figure 4 The process shown in FIG. 1 may be performed by a component of EEG calibration system 100 (e.g., calibration device 150). In other embodiments, other entities (e.g., controller 112, audio controller 250, audio system 300) may perform the calibration. Figure 4 Some or all of the steps in the embodiment. Various embodiments may include different steps and / or additional steps, or perform the steps in a different order.
[0095] The EEG calibration system 100 determines 410 that a user wears an in-ear device by placing an IED in the ear canal of the user. For example, the controller 160 may determine that the user wears the in-ear device by placing the IED in the ear canal of the user based on sensor data from one or more sensors (e.g., the motion sensor 110 of the IED 130). The calibration device 150 may be configured to perform the steps of the method 400 to perform a calibration operation each time the user wears the IED by placing the IED in the ear canal, or to perform a calibration operation each time the user otherwise determines that the IED is powered on (e.g., as part of a start sequence or startup process). Alternatively or additionally, the calibration device 150 may be configured to perform the steps of the method 400 to perform a calibration operation in response to a user operation (e.g., a user performs a predetermined operation in an application, such as on a smartphone). Alternatively or additionally, the calibration operation may be automatically performed when the calibration device 150 determines that it is necessary to perform a calibration operation (e.g., because the user has begun using a specific feature of a head mounted device or wearable device that requires accurate EEG data).
[0096] The EEG calibration system 100 presents 420 a calibration audio signal to the user via a speaker of the IED. For example, the calibration device 150 (or the controller 112) may control, for example, the speaker 102 of the IED 130 to present the calibration audio signal to the user. In one embodiment, the calibration audio signal may be a piece of music that has been embedded with predetermined audible features (e.g., a plurality of clicks or chirps or real tones known to induce neural responses in the user as auditory evoked potentials time-locked to stimulus presentation). In some embodiments, the calibration audio signal may be pre-generated and stored in a data repository (e.g., the data repository 155 of the calibration device 150). The calibration audio signal may be one of a plurality of pre-generated and stored calibration audio signals, each of which is embedded with the same or different predetermined audible features, and the EEG calibration system 100 may present 420 one or more of the stored calibration audio signals for the calibration operation based on a predetermined setting (e.g., based on a user preference or setting). In other embodiments, a calibration audio signal may be customized and dynamically generated (e.g., by the calibration device 150 or by the controller 112 of the IED 130) and presented to the user of the IED based on user data by synthesizing audio content (e.g., a recently played audio track, a track set as a favorite, other audio or sound content or tunes selectively set by the user) with predetermined audible features (e.g., chirps, clicks, and real tones, etc.). For example, clicks or tones (such as a metronome playing) may be overlaid on top of a song, where most of these tones / clicks may have a specific frequency, but other tones may be of different frequencies. The number of clicks may depend on the length of the music selection (e.g., 1 minute) and the number of beats per phrase of the music selection (e.g., 3 beats, 4 beats, etc.).
[0097] The EEG calibration system 100 (e.g., calibration device 150, controller 112) generates 430 neural signal data based on electrical signals collected from one or more electrodes (e.g., electrodes 104, 245). For example, neural signal data (e.g., EEG signal data) can be generated (e.g., estimated) by the controller 112 and / or by the controller 160 of the calibration device 150 based on the electrical signals collected by the in-ear electrode 104. In the case where the neural signal data is generated by the controller 160, the electrical signals collected by the electrode 104 of the IED 130 can be converted into digital data by the ADC of the controller 112, and the digital data can be transmitted to the controller 160 of the calibration device 150 via the network 170 to generate the neural signal data. The neural signal data can define a waveform over time that represents electrical activity occurring in the user's brain at a predetermined time after the start of a stimulus, such as a click stimulus embedded in music. The EEG calibration system 100 may also pre-process the generated neural signal data (or the electrical signals used to generate the EEG signal data) to "clean the data" and remove any noise from the real neural signal data. The EEG calibration system 100 may generate neural signal data by presenting a calibration audio signal (e.g., click music) to the user for a predetermined period of time (e.g., about 30 seconds). And while presenting the audio content, using in-ear electrodes to collect electrical signals that represent the user's time-locked neural response to, for example, the clicks present in the audio content, and using the electrical signals collected for the predetermined period of time to generate an EEG signal that represents the user's brain activity during the predetermined period of time.
[0098] The EEG calibration system 100 can analyze 440 the generated (cleansed) neural signal data. For example, the calibration module 165 can generate brain activity features from the cleansed EEG signal data and identify robust amplitude and timing information. The calibration module 165 can then compare the generated brain activity features that display amplitude and timing information with predetermined threshold features to determine whether the characteristics of the EEG signal data over time (like amplitude and timing) indicate that the user's time-locked neural response to the stimulus is as expected. For example, the calibration module 165 can be configured to determine whether the EEG signal data exhibits amplitude suppression at approximately 50ms after the onset of a click stimulus embedded in the music. The calibration module 165 can be configured to make a determination by, for example, averaging responses to multiple stimuli (e.g., multiple clicks) embedded in a calibration audio signal presented to the user for a predetermined period of time.
[0099] The calibration module 165 can perform 450 actions based on the analysis 440 of the neural signal data. Figure 5The actions performed at block 450 based on the analysis 440 of the neural signal data are explained in more detail.
[0100] Figure 5 is a flow diagram of a method 500 for determining different actions to perform based on a determined quality of neural signal data in accordance with one or more embodiments. Figure 5 The process shown in FIG. 1 may be performed by a component of the EEG calibration system 100 (e.g., the calibration device 150). In other embodiments, other entities (e.g., the controller 112 of the IED 130, the audio controller 250, the audio system 300) may perform the calibration. Figure 5 Some or all of the steps in the embodiment. Various embodiments may include different steps and / or additional steps, or perform the steps in a different order.
[0101] The calibration module 165 determines 510 that Figure 4 The block 430 generates and filters the Figure 4 The neural signal data analyzed at box 440 of satisfies predetermined amplitude conditions and time conditions. For example, the calibration module 165 determines 510 whether the electroencephalogram (EEG) data exhibits amplitude suppression at approximately 50 ms after the onset of a stimulus embedded within the calibration audio signal. That is, the calibration module 165 can measure the EEG response to repeated pairs of 50 millisecond auditory clicks separated by approximately 500 milliseconds. And in this example, the calibration module 165 can determine 510 whether there is suppression of the second click (because this is a redundant stimulus). That is, the calibration module 165 can determine 510 whether there is a threshold amplitude difference between the P50 peak of the first click stimulus and the P50 click in the second click.
[0102] In response to determining that the neural signal satisfies the predetermined amplitude condition and time condition (yes at block 510), the calibration module 165 determines that the in-ear electrode 104 is in good contact with the inner surface of the user's ear canal 118, and the generated EEG signal data is true neural data that accurately represents the user's brain activity time-locked in response to real-time stimulation. The calibration module can present a notification of this result to the user (e.g., displaying a notification on the display element 220, and instructing the speaker 102 to notify the user of the good fit of the IED in the ear canal 118, etc.).
[0103] On the other hand, in response to determining that the neural signal data does not meet the predetermined amplitude condition or time condition (no at box 510), the calibration module determines 530 whether the delayed brainstem response to the stimulus, the weak brainstem response, or both are consistent with hearing loss by performing a pitch discrimination operation to test the user's hearing. In order to eliminate the ambiguity between hearing loss and electrode function or the situation where the in-ear electrode 104 of the IED 130 is not properly installed in the ear canal 118, the calibration module can call the pitch discrimination module 180 to perform a pitch discrimination task to test the user's hearing ability.
[0104] The pitch discrimination task performed by the pitch discrimination module 180 can result in one of a positive result and a negative result. If the pitch discrimination module 180 determines that the user is able to distinguish the pitch of the audio content presented to the user during the pitch discrimination task, a positive result occurs. If the pitch discrimination module 180 determines that the user cannot distinguish the pitch of the audio content presented to the user during the pitch discrimination task, a negative result occurs. In the case of a positive result, the calibration module determines 540 that the delayed response, the weak response, or both (i.e., the amplitude and / or time conditions are not met at box 510) is due to the IED 130 not being well installed in the ear canal 118 (or an electrode failure; for example, the electrode is poorly in contact with the inner surface of the ear canal, the distance or depth of the electrode inserted into the ear canal is not enough, and the electrode needs to be replaced, etc.). The calibration module 165 can also determine whether the IED 130 is not well installed in the ear canal 118 by analyzing the EEG signal characteristics of the IED 130. That is, the EEG signal characteristics of the IED 130 that is not well installed in the ear canal 118 may be a straight line or too noisy in the neural response (where the noise looks like frequent oscillations or unshaped "ups" and "downs" that cannot be explained in the neural data). And the calibration module 165 can make a determination that the IED 130 is not well installed based on the EEG signal characteristics. The calibration module 165 can also be configured to present a notification of this result to the user (for example, displaying a notification on the display element 220 prompting the user to readjust the position of the IED in the ear canal or replace the IED, and controlling the speaker 102 to prompt the user to readjust the position of the IED in the ear canal or replace the IED, etc.).
[0105] In the case of a negative result, the calibration module 165 determines that the delayed response, the weak response, or both (i.e., the amplitude and / or time conditions are not met at block 510) is due to the user's hearing loss. The calibration module 165 can also be configured to present a notification 550 of this result to the user (e.g., instructing the speaker 102 to display a notification or displaying a notification on the display element 220 to notify the user of the hearing loss).
[0106] The calibration module can also be configured to perform signal processing (e.g., via the signal processing module 166) to compensate for the user's hearing loss by performing signal processing to correct (e.g., amplify) the gain 560 of a predetermined frequency, thereby adjusting the audio from the speaker 102 presented to the user based on the determined degree of hearing loss of the user.
[0107] Additional configuration information
[0108] The above description of the embodiments has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the patent rights to the precise form disclosed. Those skilled in the relevant art may recognize that many modifications and variations are possible in light of the above disclosure.
[0109] Some parts of this description describe embodiments in terms of algorithms and symbolic representations for operating on information. These algorithmic descriptions and representations are often used by technicians in the field of data processing to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, these operations are understood to be implemented by computer programs or equivalent circuits or microcodes, etc. In addition, without loss of generality, it is sometimes convenient to refer to the arrangement of these operations as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.
[0110] Any of the steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules alone or in combination with other devices. In one embodiment, a software module is implemented using a computer program product including a computer-readable medium, the computer-readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.
[0111] Embodiments may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the desired purpose, and / or the apparatus may include a general-purpose computing device that is selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium that can be coupled to a computer system bus, or in any type of medium suitable for storing electronic instructions. In addition, any computing system mentioned in this specification may include a single processor, or may be an architecture that employs a multi-processor design for increased computing power.
[0112] Embodiments may also be directed to a product produced by the computing process described herein. Such a product may include information obtained from the computing process, wherein the information is stored on a non-transitory tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.
[0113] Finally, the language used in this specification is selected primarily for readability and instructional purposes, and the language may not be selected to define or limit the patent rights. Therefore, it is intended that the scope of the patent rights is not limited by this specific embodiment, but is limited by any claims published on an application based on this document. Therefore, the disclosure of the embodiments is intended to illustrate rather than limit the scope of the patent rights, which is set forth in the claims.
Claims
1. A system comprising: An in-ear device (IED), the IED being configured to be placed in an ear canal of a user, the IED comprising: a speaker configured to present a calibration audio signal to the user, wherein the calibration audio signal is embedded with a predetermined audible feature, and an in-ear electrode configured to contact an inner surface of the ear canal, and A controller, the controller being configured to: instructing the speaker to present the calibration audio signal to the user, generating neural signal data based on electrical signals from the in-ear electrodes, the electrical signals corresponding to brain activity of the user in response to the predetermined audible feature, determining the quality of the generated neural signal data, and An action is performed based on a quality of the neural signal data.
2. The system according to claim 1, wherein: The electrical signal is an electroencephalogram (EEG) signal, and wherein the neural signal data is EEG signal data.
3. The system according to claim 2, further comprising a notification unit, in, The controller is configured to determine the quality of the neural signal data by determining whether the neural signal data satisfies a predetermined amplitude suppression condition and a predetermined time condition after the predetermined audible feature begins, and Wherein, the controller is configured to, in response to determining that the neural signal data satisfies the predetermined amplitude suppression condition and the predetermined time condition, instruct the notification unit to notify the user that the in-ear electrode is in contact with the inner surface of the ear canal; and preferably, wherein the controller is also configured to determine the result of the pitch discrimination operation in response to determining that the neural signal data does not satisfy the predetermined amplitude suppression condition and the predetermined time condition.
4. The system according to claim 3, wherein: The controller is further configured to, in response to determining that the result of the pitch discrimination operation is affirmative, instruct the notification unit to notify the user of one of the following: The in-ear electrode is not in contact with the inner surface of the ear canal; or The in-ear electrode is malfunctioning.
5. The system according to claim 3, wherein: The controller is further configured to, in response to determining that the result of the pitch discrimination operation is negative, instruct the notification unit to notify the user of the hearing loss; and preferably, wherein the controller is further configured to perform signal processing to correct the gain of the audio signal to compensate for the hearing loss.
6. A system according to any preceding claim, wherein: The predetermined audible feature comprises one or more clicks, and wherein the calibration audio signal comprises a music clip synthesized with the plurality of clicks; and / or preferably, the system further comprises a head mounted device, the head mounted device comprises the controller, and the head mounted device further comprises at least a second electrode, the second electrode being configured to collect a second electrical signal corresponding to a brain activity of the user in response to the predetermined audible feature, Wherein, the controller is configured to generate the neural signal data based on the electrical signal from the in-ear electrode and the second electrical signal from the second electrode.
7. A method comprising: determining that a user is wearing an in-ear device (IED) by placing the IED in an ear canal of the user, wherein the IED includes a speaker and an in-ear electrode configured to contact an inner surface of the ear canal; presenting a calibration audio signal to the user using the speaker, the calibration audio signal having a predetermined audible feature embedded therein; generating electroencephalogram (EEG) signal data based on electrical signals from the in-ear electrodes, the electrical signals corresponding to brain activity of the user in response to the predetermined audible feature; analyzing the generated EEG signal data; and Action is performed based on the analysis.
8. The method according to claim 7, in, Analyzing the EEG signal data includes determining whether the EEG signal data satisfies a predetermined amplitude suppression condition and a predetermined time condition after the onset of the predetermined auditory feature, and Wherein, the method also includes: in response to determining that the EEG signal data satisfies the predetermined amplitude suppression condition and the predetermined time condition, notifying the user that the in-ear electrode is in contact with the inner surface of the ear canal; and preferably, the method also includes: in response to determining that the EEG signal data does not satisfy the predetermined amplitude suppression condition and the predetermined time range, determining the result of the pitch discrimination operation.
9. The method according to claim 7 or 8, further comprising: Based on a positive result of the pitch discrimination operation, one of the following is performed: Notifying the user that the in-ear electrode is not in contact with the inner surface of the ear canal; or Notifying the user that the in-ear electrode is faulty; or preferably, the method further comprises: based on a negative result of the pitch discrimination operation: notifying the user of hearing loss; as well as The gain of the audio signal is corrected to compensate for the hearing loss.
10. The method according to any one of claims 7 to 9, wherein: The predetermined audible characteristic comprises one or more click sounds, and wherein the calibration audio signal comprises a piece of music synthesized with the plurality of click sounds.
11. The method according to any one of claims 7 to 10, wherein: The EEG signal data is generated based on the electrical signal from the in-ear electrode and also based on a second electrical signal, wherein the second electrical signal corresponds to the user's brain activity that is time-locked to and responsive to the predetermined audible feature, and the second electrical signal is collected by a second electrode included in a head-mounted device that the user can wear.
12. A non-transitory computer readable medium having a computer executable program stored thereon, the program comprising instructions that when executed by one or more processors cause the one or more processors to: determining that a user is wearing an in-ear device (IED) by placing the IED in an ear canal of the user, wherein: The IED includes a speaker and an in-ear electrode, the in-ear electrode being configured to contact an inner surface of the ear canal; controlling the speaker to present a calibration audio signal to the user, the calibration audio signal being embedded with a predetermined audible feature; generating neural signal data based on electrical signals from the in-ear electrodes, the electrical signals corresponding to brain activity of the user in response to the predetermined audible feature; determining the quality of the neural signal data; as well as An action is performed based on the determined quality of the neural signal data.
13. The non-transitory computer readable medium of claim 12, in, The instructions that cause the one or more processors to determine the quality of the neural signal data include instructions that, when executed by the one or more processors, cause the one or more processors to: determine whether the neural signal data satisfies a predetermined amplitude suppression condition and a predetermined time condition after the predetermined audible feature begins, and The program also includes instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations: in response to determining that the neural signal data satisfies the predetermined condition, a control notification unit notifies the user that the in-ear electrode is in contact with the inner surface of the ear canal.
14. The non-transitory computer readable medium according to claim 12 or 13, wherein: The predetermined audible characteristic comprises one or more click sounds, and wherein the calibration audio signal comprises a piece of music synthesized with the plurality of click sounds.
15. The non-transitory computer readable medium according to any one of claims 12 to 14, wherein: The instructions that cause the one or more processors to generate the neural signal data include instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: generate the neural signal data based on the electrical signal from the in-ear electrode and also based on a second electrical signal, wherein the second electrical signal corresponds to the brain activity of the user in response to the predetermined audible feature, and the second electrical signal is collected by a second electrode, and the second electrode is included in a head-mounted device that the user can wear.
Citation Information
Cited By
Hearing test method, hearing aid and hearing aid system
CN119184680A
