Hearing assistive devices with intelligent audio focus control

Abstract

A method includes: providing a smart contact lens including a beamforming microphone array, wherein the smart contact lens is worn by a user and the beamforming microphone array receives sounds from different directions; determining a gaze direction of the user; filtering the sounds received by the beamforming microphone array based on the gaze direction of the user to determine filtered sounds; and outputting the filtered sounds via a bone conduction hearing device.

Description

Background Art

[0001] The present disclosure relates generally to a hearing assistive device, and more particularly to a hearing assistive device with intelligent audio focus control.

[0002] Many spoken words are ambiguous to any listener. Often, ambiguous spoken words have similar sounds and are difficult to distinguish or understand.

[0003] In many cases, listeners use visualization of a speaker's lips to help understand or clarify the sounds of spoken words. Even in cases where visual cues are used, the interaction between hearing and vision in speech perception produces the McGurk effect, a perceptual phenomenon or illusion that occurs when the auditory component of one sound is paired with the visual component of another sound, resulting in the perception of a third sound. Summary of the invention

[0004] According to one aspect, a method is provided, comprising: determining, by a processor, a direction of sound received by a beamforming microphone array, wherein the beamforming microphone array receives sound from different directions; determining, by the processor, a focus based on the direction of the sound; outputting, by the processor, a signal based on the focus; receiving, by an audio output device, the signal output by the processor; and outputting, by the audio output device, a filtered sound, wherein the filtered sound preferentially processes certain sounds received by the beamforming microphone array based on the signal output by the processor.

[0005] According to another aspect, a non-temporary computer-readable storage medium is provided, comprising computer-executable instructions, which, when executed by a computer, cause the computer to perform a method for filtering sound, the method comprising: determining, by a processor, a direction of sound received by a beamforming microphone array, wherein the beamforming microphone array receives sound from different directions; determining, by the processor, a focus based on the direction of the sound; outputting, by the processor, a signal based on the focus; receiving, by an audio output device, the signal output by the processor; and outputting, by the audio output device, filtered sound, wherein the filtered sound preferentially processes certain sounds received by the beamforming microphone array based on the signal output by the processor.

[0006] According to another aspect, a system is provided, comprising: a beamforming microphone array; a processor that electronically communicates with the beamforming microphone array and is configured to determine a direction of sound received by the beamforming microphone array, determine a focus based on the direction of the sound, and output a signal based on the focus; and an audio output device that is configured to receive the signal output by the processor and output filtered audio, wherein the filtered audio prioritizes certain sounds received by the beamforming microphone array based on the signal output by the processor.

[0007] According to some embodiments of the present invention, a method includes: providing a smart contact lens including a beamforming microphone array, wherein the smart contact lens is worn by a user and the beamforming microphone array receives sounds from different directions; determining a gaze direction of the user; filtering the sounds received by the beamforming microphone array based on the gaze direction of the user to determine filtered sounds; and outputting the filtered sounds via a bone conduction hearing device.

[0008] According to at least one embodiment of the present invention, a system includes: a beamforming microphone array; a processor, which electronically communicates with the beamforming microphone array and is configured to determine the direction of sound received by the beamforming microphone array, determine a focus based on the direction of the sound; and output a signal based on the focus; and an audio output device, configured to receive the signal output by the processor and output filtered audio, wherein the filtered audio preferentially processes certain sounds received by the beamforming microphone array based on the signal output by the processor.

[0009] As used herein, "facilitating" an action includes performing the action, making the action easier, assisting in performing the action, or causing the action to be performed. Thus, by way of example and not limitation, instructions executing on one processor may facilitate an action performed by instructions executing on a remote processor by sending appropriate data or commands to cause or assist in performing the action. For the avoidance of doubt, where an actor facilitates an action by an action other than performing the action, the action is still performed by some entity or combination of entities.

[0010] One or more embodiments of the present invention or elements thereof can be implemented in the form of a computer program product comprising a computer-readable storage medium having a computer-usable program code for performing the indicated method steps. In addition, one or more embodiments of the present invention or elements thereof can be implemented in the form of a system (or device) comprising a memory and at least one processor, the at least one processor being coupled to the memory and operable to perform the exemplary method steps. Still further, in another aspect, one or more embodiments of the present invention or elements thereof can be implemented in the form of an apparatus for performing one or more of the method steps described herein; the apparatus can include (i) a hardware module, (ii) a software module stored in a computer-readable storage medium (or multiple such media) and implemented on a hardware processor, or (iii) a combination of (i) and (ii); any of (i)-(iii) implements the specific techniques set forth herein.

[0011] The technology of the present invention can provide substantial and beneficial technical effects. For example, one or more embodiments can provide:

[0012] A system including a hearing device configured to assist a user in understanding sounds from a surrounding environment based on a gaze direction of the user;

[0013] A system comprising a hearing device configured to assist a user in filtering sounds from a surrounding environment according to a predetermined priority; and

[0014] A system including visual simulation / augmentation capabilities corresponding to filtered sounds from the environment.

[0015] These and other features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings:

[0017] Figure 1 is a flow chart of a method of operating a hearing assistance system according to an embodiment of the present invention;

[0018] Figure 2 is a diagram of a hearing assistance system according to an embodiment of the present invention; and

[0019] Figure 3 A computer system is described that can be used to implement one or more aspects and / or elements of the present invention. DETAILED DESCRIPTION

[0020] According to some embodiments, a system is configured to help a user hear sounds from the surrounding environment, but if multiple sounds come from different directions of the environment at the same point in time, it may become confusing. One or more embodiments are directed to a system that includes a directional microphone array for analyzing directional sounds relative to a user (and relative to the user's gaze direction) and identifying the user's gaze direction and focal length within the field of view.

[0021] Some embodiments include a bone conduction hearing device that selectively enhances sounds coming from one or more directions. According to at least one embodiment, a system including a bone conduction hearing device is configured to help a user understand sounds from the surrounding environment. According to one or more embodiments of the present invention, a system uses a bone conduction hearing device configured to assist a deaf or hearing-impaired user. A bone conduction hearing device transfers sound directly to the user's cochlea through bone vibrations, bypassing the user's outer ear and middle ear. Embodiments of the present invention may extend to other types of cochlear implant hearing devices.

[0022] Embodiments of the present invention include a training method for training a system to understand the direction of sounds of interest based on the user's gaze direction. According to at least one embodiment, the method assumes a correlation between the user's gaze direction and what the user is listening to (e.g., listening to a street performance in an environment that includes nearby pedestrians and traffic noise).

[0023] Embodiments of the present invention may be implemented with smart contact lenses. It should be understood that existing contact lenses support microscale components, including, for example, single crystal silicon transistors, radio chips, antennas, diffused resistors, light emitting diodes (LEDs), silicon photodetectors, etc. These microscale components may be integrated into contact lenses, where these components are encapsulated in biocompatible polymers, etc.

[0024] Embodiments of the present invention may be extended to different types of eyewear (e.g., sunglasses, smart glasses, etc.) and other head-mounted devices and attire. For example, in some embodiments, the focus of the user is derived based on the orientation of the eyeglass frame to one or more sound sources (e.g., the system determines whether the frame is facing a particular sound source). That is, according to some embodiments, the beamforming microphone array (e.g., 211 (see Figure 2 )) is substantially directed in the direction of the view provided by the frame 210 when worn by the user, for example, disposed on a top strip of the frame 210. It should be understood that the array 211 can be configured to capture a narrow or wide sound field relative to the user's field of view (FOV). According to one or more embodiments, the eyeglass frame is configured with a beamforming microphone array (e.g., a circular microphone array including 7 (N) MEMS (micro-electromechanical system) microphones) and a processor (see Figure 316), and optionally a gaze detection camera 212. The system infers the user's FOV based on the sounds detected by the microphone array 211, and further detects the user's gaze direction within the FOV based on the sounds detected by the microphone array 211, for example, by assuming that the gaze direction is within a small cone extending away from the array 211 and the eyeglass frame 210 (relative to the FOV).

[0025] According to at least one embodiment of the present invention, where the system includes a gaze detection camera 212, the system detects the gaze direction based on the optical theory of spherical surfaces for paraxial rays and the Gullstrand model of the eye to restore the 3D center of the cornea and pupil based on an image of the user's eye captured by the camera.

[0026] According to one or more embodiments of the invention, the system further determines the direction of one or more sounds within the user's FOV. Methods for determining the direction of sounds are well known and include, for example, delay-and-sum beamforming techniques that calculate the arrival time delays of sounds received at different microphones relative to the sound arriving at a given microphone, and then superimpose time-shifted versions of the N signals.

[0027] According to one or more embodiments of the present invention, based on the user's gaze direction, the device selectively enhances the audio signal of the sound of interest using a bone conduction hearing device. According to some embodiments of the present invention, the required gaze direction can be determined by processing the audio signal of the microphone array to locate the detected sound (e.g., sound localization in three-dimensional position, including horizontal angle, vertical angle and distance (for static sound) or speed (for moving sound)). For example, when the user is looking directly at the object emitting the sound, the delay determined for the sound at each microphone in the array will be substantially equal. According to one or more embodiments, the required gaze direction can be determined based on the characteristics of the detected sound (e.g., the loudest sound in the environment, the sound with a frequency range between about 1-3 kilohertz (kHz) corresponding to many alarms (which is often located near the exit in the case of fire alarms), or a combination of thresholds, such as selecting the loudest sound in the environment for the gaze direction unless the sound is detected between 1-3 kHz, etc.). According to some embodiments, the required gaze direction can be determined based on the properties of the sound detected in the environment and a predetermined priority or priority set stored by the system in, for example, a memory of the user's mobile device (e.g., the loudest sound, the sound with certain frequencies, the sound with a source close to the user, the sound corresponding to a specific speaker, etc.). Thus, the system is configured to assist a user in understanding or perceiving sounds of interest.

[0028] According to some embodiments of the present invention, method 100 includes providing a microphone array having a beamforming microphone array (see Figure 1 , 101) of a smart device (e.g., an eyeglass frame or a contact lens). For example, the smart device includes a processor and a beamforming microphone array arranged on an outer surface of the lens, and when the user / wearer of the lens looks at the surrounding environment, the user's gaze direction is determined 102 based on the direction of sounds from the area of ​​the user's FOV. According to at least one embodiment, the FOV is explicitly determined at box 102, for example, the FOV can be determined based on the gaze direction (e.g., a predetermined horizontal arc of the user's field of view, an area measured horizontally, vertically, or diagonally at a certain distance from the user's field of view, etc.). The computing system selectively filters the sounds that are in the user's focus based on the determination of the user's gaze direction 103 (e.g., based on the properties of the sounds detected in the environment and a predetermined priority set of the sounds within the gaze direction), and the bone conduction hearing device outputs the filtered sounds 104, and in at least one embodiment, the user is not distracted by sounds that are not in the user's focus area.

[0029] According to some embodiments of the present invention, the augmented reality capabilities of the smart contact lens 105 generate appropriate mouth movement simulations or animated objects aligned with the speech that the user is listening to through the bone conduction hearing device (e.g., filtering the sound) so that the user can better understand the speech. That is, the smart contact lens 105 can be configured to display an augmented reality image (e.g., added to an existing view of the user's environment) and include a contact lens, a display unit in a central area of ​​the contact lens, and peripheral devices on the contact lens and around the display unit (e.g., a microphone array, a near field communication (NFC) antenna, a processor, etc.). According to some embodiments, the microphone of the bone conduction hearing device is controlled by determining the user's FOV.

[0030] According to some embodiments of the invention, the mouth movement simulation or augmentation (e.g., animated object) is aligned with the speech that the user is listening to through the bone conduction hearing device in playback mode 106. In the playback mode, the speed of the augmented reality visual and / or filtered sound is adjusted. For example, the animated object and speech are synchronized at a slowed speed. According to some embodiments, the speed is predetermined. According to at least one embodiment, the speed is selected by the user.

[0031] According to some embodiments of the present invention, the smart contact lens is a component of a computing system (e.g., including the smart contact lens and a paired mobile device) that analyzes (at block 103) among other things, contextual environment, user's attention to an activity or subject, distraction patterns while performing an activity, etc. In addition, the computing system identifies which directional sounds to filter out and which directional sounds to transmit to the user through the bone conduction hearing device.

[0032] According to some embodiments of the present invention, the system communicates with one or more Internet of Things (IoT) devices within wireless communication range of the computer system that provide additional information about sounds in the environment, provide additional processing resources, etc. According to one embodiment, the system receives data from an IOT smoke detector indicating that an alarm or notification issued by the smoke detector has priority over other sounds in the environment. It should be understood that an alarm or notification is a sound with a direction relative to the user / system. According to some embodiments, in the absence of data from an IOT device within wireless communication range, the system allows the bone conduction hearing device to produce ambient sounds at a first (e.g., normal) level.

[0033] According to at least one embodiment, a smart contact lens includes an antenna that communicates with a bone conduction hearing device to control the level of ambient sound output by the device, wherein the bone conduction hearing device includes a processor for analyzing sound received by a microphone array.

[0034] According to some embodiments of the invention, a computing system analyzes the context of a user's activities, environment, etc. to identify a target focus range for the user (e.g., in a boardroom) (at box 102), and identifies which directional sounds the bone conduction hearing device will output (at box 103).

[0035] Referring to block 102, according to one or more embodiments of the present invention, a current gaze direction of a user is determined based on sounds in an environment detected by a beamforming microphone array. According to some embodiments, the determination of the gaze direction 102 includes determination of a central portion of the gaze direction (e.g., a 5-degree cone), a sub-central portion of the gaze direction (e.g., an 8-degree cone), and a spot portion of the gaze direction (e.g., an 18-degree cone), such as Figure 2 As shown, for example, once the central portion of the user's current gaze direction is determined, the sub-central portion, the spot portion, etc. can be extrapolated. According to at least one embodiment, the method enhances the sound originating from one or more of these cones relative to the surrounding areas (e.g., the near peripheral area, the middle peripheral area, etc.).

[0036] Figure 2 A beamforming microphone array 201 is shown attached to a smart contact lens 202. Signals received by the microphone 201 are analyzed to determine the direction of incoming sounds from the surrounding environment. According to some embodiments, the smart contact lens 202 communicates with a bone conduction hearing device 203 to specify which directional sounds need to be filtered based on the user's focus direction 204 and the context. In addition, the system displays an augmented reality mouth 205 (or animated object) associated with the filtered sounds transmitted to the user by the bone conduction hearing device 203.

[0037] According to one or more embodiments of the present invention, microphone 201 is a piezoelectric device arranged at the periphery of contact lens 202 (e.g., outside the area of ​​the pupil). An array of piezoelectric devices (e.g., device strips) may be attached to / formed on / in a contact lens substrate. Each piezoelectric device may be mounted on a substrate that is flexible and capable of vibrating under excitation of the piezoelectric device, such as a thin polymer material including, for example, polysilicon material, silicon on insulator (SOI), or polydimethylsiloxane (PDMS). In some embodiments, the substrate is a biocompatible material that is substantially free of or free of metal. The substrate (e.g., contact lens) may be in contact with the eye. The piezoelectric device is configured to sense vibrations (e.g., sound) in the environment, thereby converting the vibrations into electrical signals. For example, a piezoelectric crystal material generates a net charge when deformed by sound.

[0038] According to one or more embodiments, the direction of the received sound is identified by the electrical generation of the piezoelectric device. For example, the system can compare the electrical generation of the device on the first side (e.g., the right portion) of the contact lens with the electrical generation of the device on the second side (e.g., the left portion) of the contact lens, where the direction of the sound is determined to be closer to the device with the higher electrical generation.

[0039] According to some embodiments, within microphone array 201, different piezoelectric devices have different resonant frequencies, for example, enabling the system to be sensitive to sounds associated with different sources (e.g., distinguishing between sound frequencies associated with conversations and sound frequencies associated with alarms).

[0040] According to one or more embodiments of the present invention, one or more of the piezoelectric devices generates electricity from mechanical forces applied during different activities and movements of a worn eye.

[0041] Figure 2 A smart contact lens 202 is shown having a beamforming microphone array 201 attached to an outer surface of the smart contact lens 202 (ie, away from the eye).

[0042] According to some embodiments, for example, the smart contact lens 202 is paired with a user's mobile device 206, wherein the smart contact lens sends wireless signals to the mobile device for analysis and control of, for example, a bone conduction hearing device.

[0043] According to some embodiments of the present invention, the bone conduction hearing device 203 is paired with the mobile device 206. According to one or more embodiments, the bone conduction hearing device 203 includes a microphone to capture sounds of the environment.

[0044] According to some embodiments of the present invention, the bone conduction hearing device 203 receives directional sound information from the smart contact lens 202 via the paired mobile device 206. For example, the smart contact lens 202 including a beamforming microphone array sends a data signal to the paired mobile device 206.

[0045] According to some embodiments of the present invention, when the user is exposed to the environment and wants to filter the unwanted sounds, the microphone 207 installed with the bone conduction hearing device 203 will be disabled and the system will use the signal received by the microphone array 201 instead. According to some embodiments, the bone conduction hearing device includes a button or switch for enabling / disabling options such as filtering. According to at least one embodiment, in the case where the user wants to listen to the sounds from the entire surrounding area, all sounds captured from the surrounding area are captured and transmitted to the user.

[0046] According to one or more embodiments, smart contact lens-based interactions are analyzed to identify that a user should turn their attention to a specific direction when performing an activity. For example, smart contact lens-based interactions are analyzed based on sounds received by a microphone array to identify that a user should turn their attention to a specific direction.

[0047] According to some embodiments of the present invention, the smart contact lens operates based on the context of the situation and the user's attention to the activity. Examples of contexts include one-to-one conversations, many-to-one conversations, business, etc. The user can focus attention on a specific activity, which allows ambient sounds in the direction of the activity to be amplified or otherwise prioritized by the system.

[0048] In accordance with one or more embodiments, the user's smart contact lenses and surrounding devices identify the angular direction (based on processing of received sounds) that the user needs to maintain attention in while performing any activity.

[0049] According to some embodiments of the present invention, a smart contact lens identifies a user's focus angle at which the user should maintain focus.

[0050] According to some embodiments of the present invention, the beamforming microphone receives sounds alone along with directional information. According to some embodiments of the present invention, the directional information is sent to the lens user's mobile device, which identifies which sounds are not in the user's focus or attention.

[0051] According to one or more embodiments, a paired mobile device filters sounds based on directional analysis and accordingly identifies which sounds are appropriate for the user's attention.

[0052] According to one or more embodiments, the mobile device communicates with the bone conduction hearing device to deliver sound based on a desired focus direction of the user (e.g., desired based on some sound in the environment, such as a siren or another sound). According to some embodiments, the user's cognitive state may be considered in determining the desired focus direction (e.g., if the user's gaze has not changed for a certain amount of time, the system may stimulate the user by calling attention to another direction). The desired focus direction may be identified by the system by identifying the user's cognitive state (e.g., from breathing patterns associated with different cognitive states) and the current focus direction (e.g., is the current focus direction away from the desired focus direction) and identifying which directional sound the bone conduction system will filter. According to some embodiments, based on the surrounding IoT data, the system identifies one or more sounds to be amplified, enhanced, or otherwise prioritized over other sounds. According to at least one embodiment, using co-hypothesis calculations, the system identifies the user's target focus range (i.e., the distance between the user and the origin location of the sound), for example, through a sound localization method, and identifies one or more sounds to be preferentially filtered, for example, based on the acoustic decibel levels of the different sounds.

[0053] According to one or more embodiments, a user hears sounds based on the user's focus area (ie, current gaze direction), and the smart contact lens identifies the directional sounds and controls what the user hears accordingly.

[0054] According to an embodiment of the present invention, the mobile device analyzes sounds from the entire environment and identifies whether any other directional sounds are also determined to have priority, such as alarms, sirens, approaching sounds, sounds approaching at an increasing speed, etc.

[0055] According to an embodiment of the invention, the mobile device analyses the sounds received by the microphone array and allows (ie filters) one or more of the sounds to be transmitted to and / or by the bone conduction hearing device.

[0056] In accordance with one or more embodiments, the system uses a beamforming microphone array to identify a user's current focus and desired attention.

[0057] According to some embodiments of the present invention, the system analyzes sensed speech patterns (of sounds received by the microphone array) and is configured to display an augmented view with synchronized lip / mouth movements. According to at least one embodiment, the system is capable of detecting or identifying different speakers through known speech separation techniques, and is capable of attaching priorities to the speakers so that the user's attention is directed to a speaker or speakers that are within the current gaze direction and have a relatively high priority to help isolate speaker voices in the environment and overcome the cocktail party effect.

[0058] According to some embodiments of the present invention, when the user is listening to selective directional sound output by a bone conduction hearing device and there is ambiguity in the sound, the system displays or projects a synchronized mouth (e.g., augmented reality) using an external display (e.g., of the mobile device 206), light generated by a light emitting diode of a smart contact lens and synchronized with the filtered sound, etc. According to one or more embodiments, the augmented reality object (e.g., a mouth or light) is a visual aid that helps the user understand what is being heard.

[0059] According to an embodiment of the present invention, the playback mode can be used to control the mouth movement simulation or animation object aligned with the sound (e.g., speech) that the user listens to using the bone conduction method. According to an embodiment of the present invention, in the playback mode, the user can adjust the playback speed so that the animation object and the speech are played synchronously at a comfortable speed (i.e., for the user).

[0060] Summary:

[0061] According to one or more embodiments of the present invention, a method includes: providing a smart contact lens (101) including a beamforming microphone array, wherein the smart contact lens is worn by a user and the beamforming microphone array receives sounds from different directions; determining a gaze direction of the user (102); filtering the sounds received by the beamforming microphone array based on the gaze direction of the user to determine filtered sounds (103); and outputting the filtered sounds via a bone conduction hearing device (104).

[0062] The methods of the embodiments of the present disclosure may be particularly suitable for use in electronic devices or alternative systems. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment or an embodiment combining software and hardware aspects, which may be collectively referred to as a "processor", "circuit", "module" or "system" in this article.

[0063] In addition, it should be noted that any method described herein may include the additional step of providing a computer system for organizing and servicing the resources of a computer system. Further, a computer program product may include a tangible computer-readable recordable storage medium having code adapted to be executed to perform one or more method steps described herein, including providing a system with different software modules.

[0064] One or more embodiments of the present invention or elements thereof can be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory and operable to perform exemplary method steps. Figure 3 A computer system that can be used to implement one or more aspects and / or elements of the present invention is shown, which also represents a cloud computing node according to an embodiment of the present invention. Figure 3 , cloud computing node 10 is only one example of a suitable cloud computing node and is not intended to impose any limitation on the scope of use or functionality of the embodiments of the invention described herein. Regardless, cloud computing node 10 is capable of implementing and / or performing any of the functions set forth above.

[0065] In the cloud computing node 10 there is a computer system / server 12, which can operate with many other general or special purpose computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations that may be suitable for the computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices, etc.

[0066] Computer system / server 12 may be described in the general context of computer system executable instructions (e.g., program modules) executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer system / server 12 may be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in local and remote computer system storage media including memory storage devices.

[0067] like Figure 3 As shown, the computer system / server 12 in the cloud computing node 10 is shown in the form of a general-purpose computing device. The components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples different system components including the system memory 28 to the processor 16.

[0068] Bus 18 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0069] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system / server 12, and includes volatile and nonvolatile media, removable and non-removable media.

[0070] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, and commonly referred to as a "hard drive"). Although not shown, a disk drive for reading from or writing to a removable non-volatile disk (e.g., a "floppy disk") may be provided, as well as an optical drive for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media). In such a case, each may be connected to the bus 18 via one or more data media interfaces. As will be further depicted and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0071] A program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 28 by way of example and not limitation. Each or some combination of the operating system, one or more application programs, other program modules, and program data may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of embodiments of the present invention as described herein.

[0072] The computer system / server 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.); and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via input / output (I / O) interface 22. In addition, the computer system / server 12 may communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via bus 18. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units and external disk drive arrays, RAID systems, tape drives, and data archiving storage systems, etc.

[0073] Thus, one or more embodiments may utilize software running on a general purpose computer or workstation. Figure 3 , such an implementation may employ, for example, a processor 16, a memory 28, and an input / output interface 22 to a display 24 and external devices 14, such as a keyboard, a pointing device, etc. The term "processor" as used herein is intended to include any processing device, for example, a processing device including a CPU (central processing unit) and / or other forms of processing circuitry. In addition, the term "processor" may refer to more than one individual processor. The term "memory" is intended to include memory associated with a processor or CPU, for example, RAM (random access memory) 30, ROM (read-only memory), fixed storage devices (e.g., hard disk drives 34), removable storage devices (e.g., disks), flash memory, etc. In addition, the phrase "input / output interface" as used herein is intended to consider, for example, an interface for one or more mechanisms (e.g., a mouse) for inputting data to a processing unit, and an interface for one or more mechanisms (e.g., a printer) for providing results associated with a processing unit. The processor 16, the memory 28, and the input / output interface 22 may be interconnected, for example, via a bus 18 as part of the data processing unit 12. Suitable interconnections (e.g., via bus 18) may also be provided to a network interface 20 (such as a network card) and a media interface (such as a floppy disk or CD-ROM drive), where the network interface 20 may be provided for connection to a computer network interface and the media interface may be provided for connection to a suitable media interface.

[0074] Thus, computer software including instructions or codes for executing the methods of the present invention described herein may be stored in one or more associated memory devices (e.g., ROM, fixed or removable memory), and when ready to be used, partially or completely loaded (e.g., loaded into RAM) and implemented by the CPU. Such software may include, but is not limited to, firmware, resident software, microcode, etc.

[0075] A data processing system suitable for storing and / or executing program code will include at least a processor 16 coupled directly or indirectly to memory elements 28 through a system bus 18. The memory elements may include local memory used during actual implementation of the program code, bulk storage, and cache memories 32 which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during implementation.

[0076] Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I / O controllers.

[0077] Network adapter 20 may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.

[0078] As used herein, including in the claims, a "server" includes a physical data processing system (e.g., a computer) running a server program. Figure 3 System 12 is shown.) It will be appreciated that such a physical server may or may not include a display and keyboard.

[0079] It should be noted that any method described herein may include the additional step of providing a system including different software modules contained on a computer-readable storage medium; these modules may include, for example, any or all appropriate elements depicted in the block diagrams and / or described herein; by way of example and not limitation, any, some or all of the modules / boxes and / or sub-modules / boxes described. The method steps may then be performed using the different software modules and / or sub-modules of the system described above executed on one or more hardware processors such as 16. Further, a computer program product may include a computer-readable storage medium having code suitable for implementation to perform one or more method steps described herein, including providing a system with different software modules.

[0080] An example of a user interface that may be employed in some cases is Hypertext Markup Language (HTML) code provided to a browser of a user's computing device by a server, etc. The HTML is parsed by a browser on the user's computing device to create a graphical user interface (GUI).

[0081] The present invention may be a system, method and / or computer program product of any possible degree of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to execute various aspects of the present invention.

[0082] Computer readable storage medium can be a tangible device that can retain and store instructions for use by instruction execution devices. Computer readable storage medium can be, for example but not limited to, electronic storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device such as punch card or a protruding structure in a groove with instructions recorded thereon, and any suitable combination of the above. Computer readable storage medium as used herein should not be interpreted as a temporary signal itself, such as radio waves or other free propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses passing through fiber optic cables) or electrical signals emitted by wires.

[0083] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0084] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming languages ​​or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can perform computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit so as to perform various aspects of the present invention.

[0085] The present invention will be described below with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer readable program instructions.

[0086] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which causes the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium in which the instructions are stored includes a manufactured product containing instructions for implementing aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0087] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in or in multiple boxes in the flowchart and / or block diagram.

[0088] The flow charts and block diagrams in the accompanying drawings show the architecture, functions and operations of the possible implementations of the system, method and computer program product according to different embodiments of the present invention. To this end, each box in the flow chart or block diagram may represent a module, segment or part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may not occur in the order marked in the figure. For example, depending on the functions involved, the two blocks shown in succession can actually be executed substantially simultaneously, or these blocks can sometimes be executed in reverse order. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0089] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the present invention. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms as well. It should also be understood that when the terms "include" and / or "comprise" are used in this specification, they specify the presence of the features, wholes, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, parts and / or combinations thereof.

[0090] All means or steps in the following claims plus corresponding structures, materials, actions, and equivalents of functional elements are intended to include any structure, material, or action for performing the functions in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for the purpose of illustration and description, but is not intended to be exhaustive or limited to the present invention in the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present invention. The embodiments are selected and described in order to best explain the principles and practical applications of the present invention, and to enable others of ordinary skill in the art to understand the various embodiments of the present invention with various modifications suitable for the intended specific use.

Claims

1. A method for filtering sound, comprising: Providing a smart contact lens including a beamforming microphone array, wherein the smart contact lens is worn by a user; determining, by a processor, a direction of sound received by the beamforming microphone array, wherein the beamforming microphone array receives sound from different directions; determining, by the processor, a focus of the user based on the direction of the sound; outputting, by the processor, a signal based on the focus; receiving, by an audio output device, the signal output by the processor; and A filtered sound is output by the audio output device, the filtered sound being determined by filtering the sound received by the beamforming microphone array based on the signal output by the processor.

2. The method according to claim 1, wherein: Determining the direction of the sound includes determining a gaze direction of a user. 3 . The method of claim 2 , comprising filtering the sound received by the beamforming microphone array based on the gaze direction of the user to determine a filtered sound.

4. The method according to claim 3, wherein: The filtered sound is output via a bone conduction hearing device.

5. The method according to claim 2, 3 or 4, wherein: The gaze direction of the user is determined based on sound localization of the sound received by the beamforming microphone array.

6. The method of claim 1, further comprising displaying a visual simulation synchronized with the filtered sound.

7. The method of claim 6, further comprising varying a playback speed of the filtered sound and the visual simulation.

8. The method according to claim 6, wherein: The visual simulation is displayed by the smart contact lens.

9. The method according to claim 6, wherein: The visual simulation is displayed on an external display.

10. The method of claim 1, further comprising displaying visual enhancements synchronized with the filtered sound.

11. The method of claim 10, further comprising varying a playback speed of the filtered sound and the visual enhancement.

12. The method according to claim 10, wherein: The visual enhancement is displayed by the smart contact lens.

13. The method according to claim 10, wherein: The visual enhancement is displayed on an external display.

14. A non-transitory computer-readable storage medium comprising computer-executable instructions, which when executed by a computer cause the computer to perform a method of filtering sound, the method comprising: Providing a smart contact lens including a beamforming microphone array, wherein the smart contact lens is worn by a user; determining, by a processor, a direction of sound received by the beamforming microphone array, wherein the beamforming microphone array receives sound from different directions; determining, by the processor, a focus of the user based on the direction of the sound; outputting, by the processor, a signal based on the focus; receiving, by an audio output device, a signal output by the processor; and A filtered sound is output by the audio output device, the filtered sound being determined by filtering the sound received by the beamforming microphone array based on the signal output by the processor.

15. The non-transitory computer-readable storage medium of claim 14, wherein: Determining the direction of the sound includes determining a gaze direction of a user.

16. The non-transitory computer-readable storage medium of claim 15, wherein: The method includes filtering the sound received by the beamforming microphone array based on the gaze direction of the user to determine a filtered sound.

17. The non-transitory computer-readable storage medium of claim 16, wherein: The filtered sound is output via a bone conduction hearing device.

18. The non-transitory computer-readable storage medium of claim 15, 16 or 17, wherein: The gaze direction of the user is determined based on sound localization of the sound received by the beamforming microphone array.

19. The non-transitory computer-readable storage medium of claim 14, further comprising displaying a visual simulation or augmentation synchronized with the filtered sound, wherein the visual simulation or augmentation is displayed by at least one of the smart contact lens and an external display.

20. The non-transitory computer-readable storage medium of claim 19, further comprising varying a playback speed of the filtered sound and the visual simulation.

21. A system for filtering sound, comprising: a beamforming microphone array disposed on a frame of a contact lens or glasses, wherein the contact lens or glasses are worn by a user; a processor in electronic communication with the beamforming microphone array and configured to determine a direction of sound received by the beamforming microphone array, determine a focus of the user based on the direction of the sound, and output a signal based on the focus; and An audio output device is configured to receive the signal output by the processor and output filtered audio determined by filtering sounds received by the beamforming microphone array based on the signal output by the processor.

22. The system of claim 21, further comprising a mobile device comprising the processor, wherein: The beamforming microphone array is disposed on the contact lens.

23. The system of claim 22, wherein: The contact lens is paired with at least one of the mobile device and the audio output device.

24. The system of claim 21, further comprising a mobile device comprising the processor, wherein: The beamforming microphone array is disposed on the frame of the glasses.

25. The system of claim 24, wherein: The beamforming microphone array is pointed substantially in the direction of the view provided by the frame of the glasses when worn by a user.

26. The system of claim 22, wherein: The beamforming microphone array is paired with at least one of the mobile device and the audio output device.

27. A computer program product comprising a computer program for implementing the method according to any one of claims 1 to 13 when the computer program is run on a computer.

Images