Hearing aid system based on visual attention and auditory electroencephalogram response

Abstract

The embodiment of the invention discloses a hearing aid system based on visual attention and auditory electroencephalogram response, and relates to the technical field of hearing aids. The system comprises an electroencephalogram collection device which comprises an electrode located on the scalp of a wearer, and the electrode is used for collecting auditory electroencephalogram signals; the eye movement tracking equipment is used for detecting the pupil size change of the wearer and evaluating the intensity and concentration degree of visual attention; and the hearing aid is internally integrated with an intelligent chip, and the intelligent chip is used for determining parameters of the hearing aid according to the auditory electroencephalogram signals and visual attention. According to the embodiment of the invention, parameters of the hearing aid can be adaptively adjusted according to actual conditions of a user and environmental changes.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of hearing aids, and in particular, to a hearing aid system based on visual attention and auditory brain response. Background Art

[0002] Currently, hearing impairment problems have received increasing attention. As a common hearing aid device, hearing aids play a crucial role in improving the quality of life of hearing-impaired people.

[0003] Traditional hearing aids usually adopt fixed parameter settings and cannot be adaptively adjusted according to the actual situation of the user and environmental changes. This results in that in complex environments, users may have difficulty clearly hearing the required sounds, affecting speech intelligibility. Summary of the Invention

[0004] Embodiments of the present invention provide a hearing aid system based on visual attention and auditory brain response to solve the above technical problems.

[0005] A hearing aid system based on visual attention and auditory brain response includes:

[0006] An electroencephalogram (EEG) acquisition device, including electrodes located on the scalp of the wearer, and the electrodes are used to collect auditory EEG signals;

[0007] An eye movement tracking device, which is used to detect changes in the pupil size of the wearer and evaluate the intensity and concentration of visual attention; and,

[0008] A hearing aid, which is internally integrated with an intelligent chip, and the intelligent chip is used to determine the parameters of the hearing aid according to the auditory EEG signals and visual attention.

[0009] Optionally, the EEG acquisition device further includes a processing chip;

[0010] After the electrodes collect the EEG signals, the processing chip filters the EEG signals and extracts the P300 waveform in the EEG signals as the auditory EEG signals.

[0011] Optionally, the intelligent chip determines the parameters of the hearing aid according to the auditory EEG signals and visual attention in the following manner:

[0012] Input the auditory EEG signals and visual attention into an embedded machine learning model to obtain the gain and noise reduction intensity of the hearing aid.

[0013] Optionally, the machine learning model includes an LSTM module, a multi-layer perceptron, and a BP module;

[0014] Embedding the auditory electroencephalogram signal and visual attention input into a machine learning model to obtain the gain and noise reduction intensity of the hearing aid, including:

[0015] Input the P300 waveform in the auditory electroencephalogram signal into the LSTM module to obtain the deep features of the P300 waveform;

[0016] Quantitatively represent the intensity and concentration of the visual attention, and input the quantitative data into a multi-layer perceptron for expansion to obtain the expanded features of the visual attention;

[0017] Fuse the deep features and expanded features through a learnable weight matrix, input them into the BP module, and obtain the gain and noise reduction intensity of the hearing aid.

[0018] Optionally, before embedding the auditory electroencephalogram signal and visual attention input into a machine learning model to obtain the gain and noise reduction intensity of the hearing aid, it further includes:

[0019] Collect the auditory electroencephalogram signals, visual attention, as well as the gain and noise reduction intensity of the hearing aid of different wearers in different scenarios to form a training sample set;

[0020] Use the training sample set to train the machine learning model so that the model is applicable to different wearers and different scenarios;

[0021] Among them, the different scenarios include different sound environments, visual stimuli, and personal preferences.

[0022] Optionally, the system further includes glasses, and a microphone array is arranged on the glasses, and the microphone array adjusts the polarity and working parameters in real time according to the direction of the wearer's turning head or gaze.

[0023] Optionally, the system further includes an APP terminal;

[0024] The APP terminal is used to respond to user operations to turn on or off the visual attention enhancement function or select different scenario modes.

[0025] Optionally, the system further includes a head-mounted display, and the head-mounted display is used to provide visual information and assist in the monitoring and analysis of visual attention.

[0026] Optionally, the intelligent chip is further used for:

[0027] Determine the target sound according to the direction and concentration of visual attention;

[0028] Strengthen the intensity of the target sound and reduce the intensity of the remaining environmental sounds.

[0029] Optionally, the data of each part is transmitted in an encrypted manner.

[0030] In summary, this embodiment proposes a hearing aid system based on visual attention and auditory brain responses, which can achieve the following beneficial effects:

[0031] 1. In terms of monitoring brain responses, this embodiment uses the method of configuring electrodes on the scalp to accurately detect the brain signals of the wearer. When there is an auditory stimulus, the brain electrical activities generated by the brain, such as components like P300, can be captured and analyzed in a timely manner, so as to deeply understand the brain's processing of sounds. Compared with traditional methods, this real-time monitoring can more accurately reflect the auditory perception state of the wearer;

[0032] 2. This embodiment adopts a visual attention enhancement mechanism. When people focus their visual attention on a certain sound source or visual information related to the sound, this embodiment can make full use of the interaction between the visual and auditory regions of the brain, greatly enhancing the perception and processing ability of the sound. This is completely different from the way that traditional hearing aids simply rely on sound processing, effectively improving the pertinence and effect of sound processing;

[0033] 3. This embodiment can achieve adaptive adjustment of hearing aid parameters. Through the intelligent chip and advanced algorithms built into the hearing aid, parameters such as gain, noise reduction intensity, and frequency response can be automatically adjusted according to the brain response and visual attention state. This adaptive adjustment can respond to the wearer's needs and environmental changes in real time, providing a clearer and more comfortable auditory experience for them;

[0034] 4. By real-time monitoring the brain response and visual attention state and making adaptive adjustments, this embodiment can effectively reduce the secondary damage to hearing caused by inappropriate sound processing and protect the residual hearing of the wearer;

[0035] 5. This embodiment can achieve personalized customization. Since the brain responses and visual attention patterns of each person are different, this embodiment can make precise parameter adjustments according to individual differences. This enables the hearing aid to better fit the unique needs of users, providing a more comfortable and natural auditory experience, thereby improving the acceptance and satisfaction of users with hearing aids;

[0036] In summary, compared with the prior art, this embodiment is no longer limited to fixed parameter settings, but can make dynamic adjustments according to the individual differences and real-time situations of the wearer. It not only improves the speech intelligibility in complex or noisy environments, but also enhances the adaptability to different environments, meets the needs of personalized customization, and effectively combines the cocktail effect and the innovative integration with glasses. Brief Description of the Drawings

[0037] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0038] Figure 1 It is a schematic diagram of a hearing aid system based on visual attention and auditory brain electrical responses provided by an embodiment of the present invention. Specific Embodiments

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope protected by the present invention.

[0040] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be construed as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0041] In the description of the present invention, it should also be noted that unless otherwise clearly defined and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0042] A hearing aid system based on visual attention and auditory brain electrical responses provided by an embodiment of the present invention intelligently personalizes the hearing aid parameters using the visual attention and auditory brain electrical responses of the wearer. To illustrate the system, the complex relationships between vision, hearing, and the brain regions are introduced first.

[0043] First, for people of different age groups, there are significant differences in the interaction between hearing and vision. For example, in adults, there is a positive correlation between visual selective attention and the ERP amplitude of auditory change detection, while this correlation is not yet mature in childhood. This means that different strategies need to be adopted for users of different age groups to optimize the performance of hearing aids.

[0044] Secondly, in terms of the functional connection and coupling of brain regions, there is a significant inverted U-shaped resonance characteristic curve between the steady-state evoked response of the visual area and the intensity of auditory noise, and there is a functional connection transmission in the visual-auditory brain regions on the side when there is unilateral auditory noise input. There is a strong coupling effect between the visual occipital area and the auditory temporal area, and it has a resonance relationship with the intensity of auditory noise. This embodiment is committed to reflecting these laws in the design and optimization of hearing aids.

[0045] In addition, the cocktail party effect in audiology shows that in complex social situations, if people focus their visual attention on a specific speaker, the brain center will actively analyze the environment, reduce the overall environmental noise, highlight the voice of the speaker being focused on, and thus improve the clarity of the sound. This is also an important basis for optimizing the hearing aid system in this embodiment.

[0046] Based on the above content, Figure 1 is a schematic diagram of a hearing aid system based on visual attention and auditory EEG response provided by an embodiment of the present invention. As Figure 1 shown, the system includes an EEG acquisition device, an eye movement tracking device, a hearing aid, glasses, and an APP terminal (not shown in the figure).

[0047] Among them, the EEG acquisition device includes electrodes located on the scalp of the wearer and a processing chip. The electrodes are used to collect EEG signals, and the processing chip is used to process the collected EEG signals and extract auditory-related waveforms. Optionally, highly sensitive electrodes can be used to ensure that the EEG signals of the wearer can be accurately and stably obtained. These electrodes are arranged at specific positions on the scalp to capture the EEG activities related to auditory processing to the greatest extent. After the original EEG signals are collected, filtering operations are performed by applying complex digital signal processing algorithms to remove possible interference and noise, such as electromagnetic interference from peripheral electronic devices or physiological noise of the wearer himself. Then, denoising processing is carried out to further improve the purity of the signals so that subsequent feature extraction work can be more accurate. Then, EEG components closely related to auditory perception, such as P300, are identified using embedded algorithms.

[0048] The eye-tracking device and the head-mounted display jointly complete the visual attention detection function in the system. Among them, the eye-tracking device can accurately monitor the gaze direction of the wearer, and capture the changes in its visual focus in real time; at the same time, by monitoring the changes in pupil size, it can evaluate the intensity and concentration of visual attention. The head-mounted display can not only provide visual information, but also assist in the monitoring and analysis of visual attention.

[0049] The hearing aid is internally integrated with an intelligent chip, and the intelligent chip is wirelessly connected to the electroencephalogram acquisition device, the eye-tracking device and the head-mounted display, and can determine the parameters of the hearing aid according to the auditory electroencephalogram signal and visual attention. To achieve this function, this embodiment provides a method for automatically determining the parameters of a hearing aid, which is embedded in the intelligent chip of the hearing aid. During actual use, the hearing aid can quickly and accurately calculate the optimal parameter settings according to the real-time collected electroencephalogram and visual attention information, such as adjusting the gain to enhance the sound intensity in a specific frequency range, dynamically adjusting the noise reduction intensity to adapt to changes in environmental noise, etc.

[0050] Optionally, powerful machine learning algorithms such as neural networks and support vector machines can be used to establish a complex mapping relationship between electroencephalogram responses, visual attention and hearing aid parameters. In the training stage, by collecting a large amount of data of different individuals in various scenarios, including different sound environments, visual stimuli and personal preferences, etc., the algorithm is repeatedly trained and optimized.

[0051] In a specific embodiment, the machine learning algorithm is implemented by a neural network including an LSTM module, a multi-layer perceptron and a BP module. Among them, the LSTM module adopts the backbone structure of the LSTM model; the output dimensions of each layer of the multi-layer perceptron gradually increase to expand the feature dimensions; the BP module adopts the complete BP network structure, and finally outputs the parameters of the hearing aid to be configured. During the data processing:

[0052] First, input the P300 waveform in the auditory electroencephalogram signal into the LSTM module to obtain the deep features of the P300 waveform. Optionally, the features of the layer before the output layer of the LSTM module can be extracted as the deep features of the P300 waveform.

[0053] At the same time, the intensity and concentration of the visual attention are quantitatively represented, and the quantitative data is input into the multi-layer perceptron for expansion to obtain the expanded features of the visual attention. Among them, the intensity and concentration of visual attention can be divided into different levels, such as strong, medium, weak, etc., and different data are used to quantitatively represent different levels, such as 1, 2, 3. Input the quantitative data into the multi-layer perceptron to obtain a high-dimensional feature.

[0054] Then, the depth features and the extended features are fused through a learnable weight matrix and input into the BP module to obtain the gain and noise reduction intensity of the hearing aid. Optionally, assuming that the depth feature of the P300 waveform is a 1×M-dimensional vector and the extended feature of visual attention is a 1×N-dimensional vector, the following can be done: convert the depth feature into a 1×Q-dimensional vector A1 through an M×Q-dimensional feature matrix F1; convert the extended feature into a 1×Q-dimensional vector A2 through an N×Q-dimensional feature matrix F2; A1 + A2 is the fused feature vector. Among them, F1 and F2 are learnable weight matrices, and the specific element values are determined during model training.

[0055] In addition, in this embodiment, the hearing aid is combined with glasses, and a microphone array is arranged on the glasses. These microphones can adjust the polarity and working parameters in real time according to the direction of the wearer's head turning or gaze. For example, when the wearer looks straight ahead and focuses on someone's speech, the microphones in the front will enhance their sensitivity, while reducing the gain of the microphones in other directions, thereby effectively highlighting the target sound and improving the detection distance and signal-to-noise ratio.

[0056] At the same time, the system can also include an APP side. The user can turn on or off the visual attention enhancement function or select different scene modes through the APP side. Specifically, the software side adopts an intuitive and easy-to-use user interface. The wearer can control some basic functions of the hearing aid through simple operations, such as touching the screen or buttons, such as turning on or off the visual attention enhancement function, or selecting different preset modes to adapt to specific scenarios, such as meeting mode, outdoor mode, etc.

[0057] In addition, this hearing aid system fully combines the cocktail party effect in audiology. When the wearer locks the speaker through visual attention, the hearing aid can use the changes in the electroencephalogram response to intelligently reduce environmental noise and highlight the speaker's voice, further improving the clarity and intelligibility of the sound.

[0058] Furthermore, in the overall architecture design of the hearing aid, this embodiment adopts a compact, lightweight and comfortable shape to ensure that the wearer will not feel discomfort during long-term use. At the same time, the internal hardware components, including the intelligent chip, audio processing unit and power management system, etc., are carefully optimized and integrated to achieve efficient performance and low-power operation.

[0059] In addition, in order to ensure data security and privacy protection, this embodiment adopts advanced encryption technology to encrypt the collected electroencephalogram and visual data for transmission and storage, and only authorized personnel or devices can access and process this sensitive information.

[0060] It should be noted that all user data involved in this application are information and data that have been authorized by users or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards in relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0061] In summary, this embodiment proposes a hearing aid system based on visual attention and auditory brain responses, which can achieve the following beneficial effects:

[0062] 1. In terms of monitoring brain responses, this embodiment uses the method of configuring electrodes on the scalp to accurately detect the brain signals of the wearer. When there is an auditory stimulus, the brain electrical activities generated by the brain, such as components like P300, can be captured and analyzed in a timely manner, so as to deeply understand how the brain processes sounds. Compared with traditional methods, this real-time monitoring can more accurately reflect the auditory perception state of the wearer;

[0063] 2. This embodiment adopts a visual attention enhancement mechanism. When people focus their visual attention on a certain sound source or visual information related to the sound, this embodiment can make full use of the interaction between the visual and auditory regions of the brain, greatly enhancing the perception and processing ability of that sound. This is completely different from the way traditional hearing aids simply rely on sound processing, effectively improving the pertinence and effect of sound processing;

[0064] 3. This embodiment can achieve adaptive adjustment of hearing aid parameters. Through the intelligent chip and advanced algorithms built into the hearing aid, parameters such as gain, noise reduction intensity, and frequency response can be automatically adjusted according to brain responses and visual attention states. This adaptive adjustment can respond in real time to the needs of the wearer and environmental changes, providing a clearer and more comfortable auditory experience for them;

[0065] 4. By real-time monitoring brain responses and visual attention states and making adaptive adjustments, this embodiment can effectively reduce the secondary damage to hearing caused by inappropriate sound processing and protect the residual hearing of the wearer;

[0066] 5. This embodiment can achieve personalized customization. Since the brain responses and visual attention patterns of each person are different, this embodiment can make precise parameter adjustments according to individual differences. This enables the hearing aid to better fit the unique needs of users, providing a more comfortable and natural auditory experience, thereby improving the acceptance and satisfaction of users with hearing aids;

[0067] 6. This embodiment also combines the hearing aid with glasses, and a multi-microphone array is set on the glasses. When the wearer turns their head or gazes in a specific direction, the polarity of the microphone in the corresponding direction is strengthened, increasing the detection distance and improving the signal-to-noise ratio, so as to more effectively capture the target sound;

[0068] 7. In a noisy environment, when the wearer focuses their visual attention on the speaker, the hearing aid can enhance the amplification and noise reduction of the speech sound, greatly improving speech intelligibility. This is a huge improvement for those who have difficulty communicating in complex auditory scenarios, enabling them to participate in conversations more smoothly and reducing communication barriers;

[0069] 8. This embodiment can accurately judge the environment based on visual information. Whether in different scenarios such as a cinema, a meeting room, or outdoors, it can automatically adjust parameters to provide a more suitable auditory effect. This means that the wearer does not need to manually adjust the settings frequently, saving time and effort, and also avoiding the inconvenience caused by improper operation.

[0070] In summary, compared with the prior art, this embodiment is no longer limited to fixed parameter settings, but can be dynamically adjusted according to the individual differences and real-time situations of the wearer. It not only improves speech intelligibility in a noisy environment, but also enhances the adaptability to different environments, meets the needs of personalized customization, and effectively combines the cocktail party effect and the innovative integration with glasses.

[0071] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A hearing aid system based on visual attention and auditory brain responses, characterized in that, Comprising: An electroencephalogram (EEG) acquisition device, including electrodes located on the scalp of the wearer, and the electrodes are used to acquire auditory EEG signals; An eye movement tracking device, used to detect changes in the pupil size of the wearer and evaluate the intensity and concentration of visual attention; And, A hearing aid, internally integrated with a smart chip, and the smart chip is used to determine the parameters of the hearing aid according to the auditory EEG signals and visual attention.

2. The system according to claim 1, characterized in that The EEG acquisition device further includes a processing chip; After the electrodes acquire the EEG signals, the processing chip filters the EEG signals and extracts the P300 waveform in the EEG signals as the auditory EEG signals.

3. The system according to claim 1, wherein The smart chip determines the parameters of the hearing aid according to the auditory EEG signals and visual attention in the following manner: Input the auditory EEG signals and visual attention into the embedded machine learning model to obtain the gain and noise reduction intensity of the hearing aid.

4. The system according to claim 3, wherein The machine learning model includes an LSTM module, a multi-layer perceptron, and a BP module; The step of inputting the auditory EEG signals and visual attention into the embedded machine learning model to obtain the gain and noise reduction intensity of the hearing aid includes: Input the P300 waveform in the auditory EEG signals into the LSTM module to obtain the deep features of the P300 waveform; Quantitatively represent the intensity and concentration of the visual attention, and input the quantified data into the multi-layer perceptron for expansion to obtain the expanded features of the visual attention; Fuse the deep features and the expanded features through a learnable weight matrix, input them into the BP module, and obtain the gain and noise reduction intensity of the hearing aid.

5. The system according to claim 4, wherein Before inputting the auditory EEG signals and visual attention into the embedded machine learning model to obtain the gain and noise reduction intensity of the hearing aid, it further includes: Collect the auditory EEG signals, visual attention, as well as the gain and noise reduction intensity of the hearing aid of different wearers in different scenarios to form a training sample set; Use the training sample set to train the machine learning model to make the model applicable to different wearers and different scenarios; Wherein, the different scenarios include different sound environments, visual stimuli, and personal preferences.

6. The system according to claim 1, wherein It further includes glasses, and a microphone array is arranged on the glasses, and the microphone array adjusts the polarity and working parameters in real time according to the direction of the wearer's head turning or gaze.

7. The system according to claim 1, wherein It further includes an APP terminal; The APP terminal is used to respond to user operations to turn on or off the visual attention enhancement function or select different scenario modes.

8. The system according to claim 1, wherein It further includes a head-mounted display, and the head-mounted display is used to provide visual information and assist in the monitoring and analysis of visual attention.

9. The system according to claim 1, wherein The smart chip is further used for: Determine the target sound according to the direction and concentration of visual attention; Strengthen the intensity of the target sound and reduce the intensity of the remaining environmental sounds.

10. The system according to claim 1, characterized in that, The data of each part is transmitted in an encrypted manner.

Images