Assistive hearing device with smart audio focus control

Abstract

Assistive hearing device with smart audio focus control [Solution] The method includes providing a smart contact lens including an array of beamforming microphones, wherein the smart contact lens is worn by a user and the array of beamforming microphones receives sounds from different directions, determining the user's gaze direction, filtering the sounds received by the array of beamforming microphones based on the user's gaze direction to determine the sounds to be filtered, and outputting the filtered sounds via a bone conduction hearing device.

Description

[Technical field]

[0001] The present disclosure relates generally to hearing assistive devices, and more particularly to hearing assistive devices with smart audio focus control. [Background technology]

[0002] There are many spoken words that create ambiguity for any listener. Typically, spoken words that create ambiguity have similar sounds that are difficult to distinguish or understand.

[0003] In many cases, listeners use the visualization of a speaker's lips to help understand or disambiguate the sounds of a spoken word. Even when using visualized cues, the interplay between hearing and vision in speech perception can give rise to the McGurk effect, where an acoustic component of one sound is paired with a visual component of another sound, inducing the perception of a third sound, resulting in a recognition phenomenon, or misinterpretation. Summary of the Invention

[0004] According to one aspect, a method is provided that includes determining, by a processor, a direction of sound received by an array of beamforming microphones receiving sound from different directions; determining, by a processor, a focus based on the direction of the sound; outputting, by a processor, a signal based on the focus; receiving, by an audio output device, the signal output by the processor; and preferentially processing, by the audio output device, certain sounds received by the array of beamforming microphones based on the signal output by the processor to output a filtered sound.

[0005] According to another aspect, there is provided a non-transient computer readable storage medium including computer executable instructions that, when executed by a computer, cause the computer to perform a method of filtering sound, the method including: determining, by a processor, a direction of sound received by an array of beamforming microphones receiving sound from different directions; determining, by a processor, a focus based on the direction of the sound; outputting, by a processor, a signal based on the focus; receiving, by an audio output device, the signal output by the processor; and outputting, by the audio output device, a filtered sound that has been preferentially processed from a sound received by the array of beamforming microphones based on the signal output by the processor.

[0006] According to another aspect, a system is provided that includes an array of beamforming microphones; a processor in electronic communication with the array of beamforming microphones and configured to determine a direction of sound received by the array of beamforming microphones, determine a focus based on the direction of sound, and output a signal based on the focus; and an audio output device configured to receive the signal output by the processor and output filtered audio, the filtered audio configured to preferentially process certain sounds received by the beamforming microphones based on the signal output by the processor.

[0007] According to some embodiments of the present invention, a method includes providing a smart contact lens including the array of beamforming microphones, wherein the smart contact lens is worn by a user and the array of beamforming microphones receives sounds from different directions and determines a gaze direction of the user, filters the sounds received by the array of beamforming microphones based on the gaze direction of the user to determine filtered sounds, and outputting the filtered sounds via a bone conduction hearing device.

[0008] In accordance with at least one embodiment of the present invention, a system includes an array of beamforming microphones; a processor in electronic communication with the array of beamforming microphones and configured to determine a direction of sound received by the array of beamforming microphones, determine a focus based on the direction of sound, and output a signal based on the focus; and an audio output device configured to receive the signal output by the processor and output filtered audio, the filtered audio configured to preferentially process certain sounds received by the beamforming microphones based on the signal output by the processor.

[0009] As used herein, the act of "facilitating" includes performing an action, facilitating an action, assisting in the performance of an action, or causing an action to be performed. Thus, by way of example, and not by way of limitation, instructions executing on one processor may facilitate an action by sending appropriate data or commands that cause or assist in the execution of instructions on a remote processor. For the avoidance of doubt, when a human operator facilitates an action by doing something other than performing the action, the action is nevertheless performed by some entity or combination of entities.

[0010] One or more embodiments of the present invention, or elements thereof, may be implemented in the form of a computer program product including a computer-readable recording medium having computer usable program code for performing the indicated method steps. Furthermore, one or more embodiments of the present invention, or elements thereof, may be implemented in the form of a system (or apparatus) including a memory and at least one processor coupled to the memory and operable to perform the exemplary method steps. Still further, in another aspect, one or more embodiments of the present invention, or elements thereof, may be implemented in the form of a means for performing one or more of the method steps described herein; the means may include (i) a hardware module(s), (ii) a software module(s) stored in a computer-readable recording medium (or a number of such media) and implemented on a hardware processor, or (iii) a combination of (i) and (ii); any of (i)-(iii) implementing the specific techniques described herein.

[0011] The techniques of the present invention can provide substantial and valuable technical advantages. For example, one or more embodiments may: A system including a hearing device configured to assist a user in understanding sounds from a surrounding environment based on a gaze direction of the user; A system including a hearing device configured to assist a user in filtering sounds from a surrounding environment according to predetermined priorities; A system is provided that includes visual simulation / augmentation capabilities in response to filtered sounds from the environment.

[0012] These and other features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

[0013] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a flow diagram of a method of operating a hearing assistance system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram of a hearing assistance system according to an embodiment of the present invention. [Diagram 3] FIG. 3 illustrates a computer system useful for implementing one or more aspects or elements of the present invention, or a combination thereof. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] According to some embodiments, the system is configured to assist a user in hearing sounds from the surrounding environment, which can become confusing when multiple sounds come simultaneously from different directions in the surrounding environment. One or more embodiments are directed to a system that includes an array of directional microphones that are used to analyze directional sounds relative to the user (and relative to the user's gaze direction) and identify focus distances within the user's gaze direction and field of view.

[0016] Some embodiments include a bone conduction hearing device that selectively enhances sound from one or more directions. According to at least one embodiment, a system including a bone conduction hearing device is configured to assist a user in understanding sounds from their surrounding environment. According to one or more embodiments of the present invention, the system uses a bone conduction hearing device configured to assist a user who is deaf or hearing impaired. A bone conduction hearing device transmits sound directly to the user's cochlea via bone vibrations, bypassing the user's outer and middle ear. Embodiments of the present invention can be extended to other types of cochlear implant hearing devices.

[0017] Embodiments of the invention include a training method that is used to train the system to understand the direction of sounds of interest based on the user's gaze direction. According to at least one embodiment, the method assumes a correlation between the user's gaze direction and what the user is hearing (e.g., listening to a street performance in an environment that includes nearby pedestrian and traffic noise).

[0018] Embodiments of the present invention can be implemented into smart contact lenses. Existing contact lenses should be understood to support microscale components including, for example, single crystal silicon transistors, radio chips, antennas, diffusion resistors, light emitting diodes (LEDs), silicon photodetectors, etc. These microscale components can be integrated into contact lenses, where the components are encapsulated in biocompatible polymers, etc.

[0019] The embodiments of the present invention extend to different types of safety glasses (e.g., sunglasses, smart glasses, etc.) and other head-mounted devices and clothing. For example, in some embodiments, the user's focus is derived based on the frame of the safety glasses and its placement on one or more sound sources (e.g., the system determines whether the frame is placed toward a particular source). That is, according to some embodiments, an array of beamforming microphones, e.g., 211 (see FIG. 2), disposed across the top bar of the frame 210, is substantially oriented in the direction of the field of view provided by the frame 210 when worn by the user. The array 211 can be configured to capture sounds in a narrow or wide range relative to the user's field of view (FOV). According to one or more embodiments, the frame of the glasses is configured with an array of beamforming microphones (e.g., 7(N) MEMS (micro-electromechanical system) microphones) and a processor (see FIG. 3, 16), and optionally a line-of-sight camera 212. The system estimates the user's FOV based on sounds detected by the microphone array 211, and further detects the user's gaze direction within the FOV detected by the microphone array 211, for example by assuming that the gaze direction is within a narrow cone (relative to the FOV) extending away from the array 211 and frame 210 of the safety glasses.

[0020] According to at least one embodiment of the invention, in cases where the system includes a gaze direction camera 212, the system detects gaze direction based on an image of the user's eye(s) captured by the camera, based on spherical optics theory for paraxial rays, and the Gullstrand model of the eye to recover the 3D centers of the cornea and pupil.

[0021] According to one or more embodiments of the present invention, the system further determines the direction of one or more sounds within the user's FOV. Methods for determining the direction of a sound are well known and include, for example, delay-and-sum beamforming techniques, which calculate the time-delayed arrival of sounds arriving at a different microphone relative to a sound received at a given microphone, and then superimpose time-shifted variants of the N signals.

[0022] According to one or more embodiments of the present invention, based on the user's gaze direction, the device selectively enhances the audio signal of the sound of interest using a bone conduction hearing device. According to some embodiments of the present invention, the requested gaze direction can be determined according to the characteristics of the detected sound (e.g., the loudest sounds in the environment, sounds having a frequency range between about 1-3 kilohertz (kHz), which corresponds to many alarms, such as fire alarms, etc.), which are determined by processing the audio signals of the microphone array to localize the detected sound (e.g., sound source localization with respect to three-dimensional position includes horizontal angle, vertical angle, and distance (for static sounds) or velocity (for moving sounds). For example, if the user is looking directly at the object generating the sound, the delays determined for the sound at each microphone of the array are substantially equal). According to one or more embodiments, the requested gaze direction can be determined according to the characteristics of the detected sound (e.g., the loudest sounds in the environment, sounds having a frequency range between about 1-3 kilohertz (kHz), which corresponds to many alarms, such as fire alarms, etc.). (e.g., a combination of thresholds such as selecting the loudest sound in the environment for the gaze direction if no sound is detected between 1-3 kHz, or selecting the loudest sound in the environment for the gaze direction if no sound is detected between 1-3 kHz). According to some embodiments, the requested gaze direction can be determined according to characteristics of sounds detected in the environment and a pre-determined priority or set of priorities stored by the system, e.g., in the memory of the user's mobile device (e.g., the loudest sounds, sounds of a certain frequency, sounds with a source approaching the user, sounds corresponding to a particular speaker, etc.). Thus, the system is configured to assist the user in understanding or recognizing sounds of interest.

[0023] According to some embodiments of the invention, method 100 includes providing a smart device (e.g., a safety eyewear frame, or a contact lens) having an array of beamforming microphones (see FIG. 1, 101). For example, the smart device may include a processor and an array of beamforming microphones disposed on an outer surface of the lens, and when a user / wearer of the lens looks around, a gaze direction of the user is determined at 102 based on a direction of sound coming from an area of ​​the user's FOV. According to at least one embodiment, the FOV is explicitly determined at block 102, for example, the FOV can be determined based on the gaze direction (e.g., a pre-determined horizontal arc of a given user's viewpoint, a field measured horizontally, vertically, or diagonally some distance away from the user's viewpoint, etc.). The computing system selectively filters sounds within the user's focus at 103 based on a determination of the user's gaze direction (e.g., characteristics of sounds detected in the environment and a pre-determined set of priorities for sounds within the gaze direction) and the bone conduction hearing device outputs the filtered sounds at 104, and in at least one embodiment, the user will not be confused by sounds that are not within the user's focus area.

[0024] According to some embodiments of the present invention, the augmented reality capabilities of the smart contact lens at 105 generate appropriate simulated mouth movements or animated objects aligned with the voice (e.g., filtered sound) the user is hearing through the bone conduction hearing device so that the user can better understand the voice. That is, the smart contact lens at 105 can be configured to display augmented reality images (e.g., adding to the user's existing field of view of the environment) and includes a central region of the contact lens and peripheral devices (e.g., microphone arrays, Near Field Communication (NFC) antennas, processors, etc.) on and around the contact lens and a display unit. According to some embodiments, the microphones of the bone conduction hearing device are controlled by a determination of the user's FOV.

[0025] According to some embodiments of the present invention, the simulated mouth movements or augmentation (i.e., animated objects) can be aligned in playback mode at 106 to the voice the user is hearing through the bone conduction hearing device. In playback mode, the speed of the augmented reality view, or the filtered sound, or both, is adjusted. For example, the animated objects and voice are synchronized at a reduced speed. According to some embodiments, the speed is pre-determined. According to at least one embodiment, the speed is selected by the user.

[0026] According to some embodiments of the present invention, the smart contact lens is a component of a user's computing system (e.g., including the smart contact lens and a paired mobile device) that analyzes (block 103), among other things, the contextual environment, the user's attention to some activity or subject, patterns of distraction while performing an activity, etc. Additionally, the computing system determines which directional sounds to filter out and which directional sounds to transmit to the user through the bone conduction hearing device.

[0027] According to some embodiments of the present invention, the system communicates with one or more Internet of Things (IoT) devices within wireless communication range of the computer system, which provide additional information regarding sounds in the environment and provide additional processing resources. According to one embodiment, the system receives data from an IoT smoke detector indicating an alarm or notification issued by the smoke detector has priority over other sounds in the environment. It should be understood that an alarm or notification is a sound that has a directionality relative to the user / system. According to some embodiments, in the absence of data from an IoT device within wireless communication range, the system allows the environmental sounds generated by the bone conduction hearing device at a first (e.g., normal) level.

[0028] According to at least one embodiment, the smart contact lens includes an antenna in communication with a bone conduction hearing device to control the level of ambient sound output by the device, where the bone conduction hearing device includes a processor for analyzing sound received by the microphone array.

[0029] According to some embodiments of the present invention, the computing system analyzes the user's activity, surroundings, and other contextual circumstances to identify the user's targeted focus area (e.g., within a board in a meeting room) (at block 102) and to identify which directional sound to output via the bone conduction hearing device (at block 103).

[0030] Referring to block 102, according to one or more embodiments of the present invention, a user's current gaze direction is determined according to sounds in the environment detected by an array of beamforming microphones. According to some embodiments, determining the gaze direction 102 includes determining a central portion of the gaze direction (e.g., a 5° cone), a paracentral portion of the gaze direction (e.g., an 8° cone), and a patchy portion of the gaze direction (e.g., an 18° cone), as shown in FIG. 2. For example, once the central portion of the user's current gaze direction is determined, the paracentral, patchy portions, etc. can be extrapolated. According to at least one embodiment, the method enhances sounds originating from within one or more of these cones relative to the surrounding regions (e.g., near surrounding region, medium surrounding region, etc.).

[0031] 2 shows an array of beamforming microphones 201 attached to a smart contact lens 202. Signals received by the microphones 201 are analyzed to determine the direction of incoming sound from the surrounding environment. According to some embodiments, the smart contact lens 202 communicates with a bone conduction hearing device 203 to identify which direction of sound needs to be filtered based on the user's focus direction 204 and contextual circumstances. Additionally, the system displays an augmented reality mouth 205 (or animated object) that correlates to the filtered sound transmitted to the user by the bone conduction hearing device 203.

[0032] According to one or more embodiments of the present invention, the microphone 201 is a piezoelectric device located at the boundary of the contact lens 202 (e.g., outside the pupil area). An array of piezoelectric devices (e.g., a strip of devices) can be attached to / formed on / within the support of the contact lens. Each piezoelectric device can be mounted on a support that is flexible and vibrates upon stimulation of the piezoelectric device, e.g., a thin polymeric material including polysilicone material, silicon-on-insulator (SOI), or polydimethylsiloxane (PDMS). In some embodiments, the support can be a substantially metal-free or free biocompatible material. The support, such as a contact lens, can be in contact with the eye. The piezoelectric device is configured to detect vibrations (e.g., sound) in the environment and convert the vibrations into an electrical signal. For example, a piezoelectric crystal material generates a net electrical charge when deformed by sound.

[0033] According to one or more embodiments, the direction of the received sound is identified by the power generation of the piezoelectric device. For example, the system compares the power generation of a device on a first side (e.g., the right portion) of the contact lens with the power generation of a device on a second side (e.g., the left portion) of the contact lens, where the direction of the sound is determined to be closer to the device having a higher power generation.

[0034] According to some embodiments, within the array of microphones 201, different piezoelectric devices may have different resonant frequencies, for example, enabling the system to be sensitive to sounds associated with different sources (e.g., distinguishing between sound frequencies associated with speech and those associated with alarms).

[0035] According to one or more embodiments of the present invention, one or more piezoelectric devices generate electrical power from mechanical forces applied during various activities and movements of the wearer's eyes.

[0036] FIG. 2 shows a smart contact lens 202 having an array of beamforming microphones 201 attached to the outer surface of the smart contact lens 202 (i.e., away from the eye).

[0037] According to some embodiments, the smart contact lenses 202 are paired with a user's mobile device 206, where, for example, the smart contact lenses send wireless signals to the mobile device for analysis and control of, for example, a bone conduction hearing device.

[0038] According to some embodiments of the present invention, a bone conduction hearing device 203 is paired with a mobile device 206. According to one or more embodiments, the bone conduction hearing device 203 includes a microphone that captures ambient sounds.

[0039] According to some embodiments of the present invention, the bone conduction hearing device 203 receives directional sound information from the smart contact lens 202 via a paired mobile device 206. For example, the smart contact lens 202, which includes an array of beamforming microphones, transmits data signals to the paired mobile device 206.

[0040] According to some embodiments of the present invention, when a user is exposed to an environment and wants to filter out unwanted sounds, the microphone 207 mounted in the bone conduction hearing device 203 is disabled and the system uses the signals received by the microphone array 201 instead. According to some embodiments, the bone conduction hearing device includes a button or switch for enabling / disabling options such as filtering. According to at least one embodiment, when a user wants to hear sounds from the entire surrounding area, all sounds captured from the surrounding area are captured and transmitted to the user.

[0041] According to one or more embodiments, smart contact lens based interactions are analyzed to identify a direction in which the user should orient and attend while performing an activity. For example, smart contact lens based interactions are analyzed based on sounds received by a microphone array to identify a direction in which the user should orient and attend while performing an activity.

[0042] According to some embodiments of the present invention, smart contact lenses operate based on a user's attention to contextual surroundings and activities. Examples of context include one-to-one conversations, many-to-one conversations, traffic, etc. A user can focus attention on a particular activity, which allows the system to amplify or otherwise preferentially process ambient sounds in the direction of the activity.

[0043] According to one or more embodiments, the user's smart contact lenses and peripheral devices identify (based on processing of received sounds) the angular direction in which the user needs to maintain attention while any activity is being performed.

[0044] According to some embodiments of the present invention, the smart contact lens identifies the focus angle at which the user should maintain his focus.

[0045] According to some embodiments of the present invention, beamforming microphones receive sounds individually along with directional information, which is then sent to a lens user's mobile device, allowing the mobile device to identify which sounds are outside of the user's focus or attention.

[0046] According to one or more embodiments, a paired mobile device filters sounds based on directional analysis and adaptively identifies which sounds are appropriate for a user's attention.

[0047] According to one or more embodiments, the mobile device communicates with the bone conduction hearing device to provide sounds based on a requested focus direction of the user (e.g., requested based on some sound in the environment, such as an alarm or another voice). According to some embodiments, the cognitive state of the user can be taken into account in determining the requested focus direction (e.g., if the user's gaze has not changed for some amount of time, the system can stimulate the user by calling attention to another direction). The requested focus direction can be identified by the system by identifying the user's cognitive state (e.g., from breathing patterns associated with different cognitive states) and a current focus direction (e.g., a current focus direction away from the requested focus direction), and by identifying which directional sounds should be filtered by the bone conduction hearing device. According to some embodiments, based on ambient IoT data, the system identifies one or more sounds that should be amplified, boosted, or otherwise prioritized over other sounds. According to at least one embodiment, using cognitive computing, the system identifies a user's target focus area by, for example, sound source localization techniques (i.e., the distance between the user and the sound source location) and identifies one or more sounds to be preferentially filtered based, for example, on the acoustic decibel levels of different sounds.

[0048] According to one or more embodiments, the user hears sounds based on the user's focus area (i.e., current gaze direction) and the smart contact lenses identify directional sounds and adaptively control what the user hears.

[0049] According to an embodiment of the invention, the mobile device analyzes sounds from the entire surroundings and identifies whether it should also determine whether any other directional sounds should have priority, such as alarms, sirens, approaching sounds, sounds approaching at increasing speeds, etc.

[0050] According to an embodiment of the present invention, the mobile device analyzes sounds received by the microphone array and allows (i.e. filters) one or more sounds to be communicated to or from the bone conduction hearing device.

[0051] According to one or more embodiments, the system uses an array of beamforming microphones to identify the user's current focus and required attention.

[0052] According to some embodiments of the invention, the system is configured to analyze detected speech patterns (of sounds received by the microphone array) and display an augmented field of view with synchronized lip / mouth movements. According to at least one embodiment, the system is able to detect or identify different speakers and prioritize speakers by known speech separation techniques so that the user's attention is directed to a relatively high priority speaker or speakers within the current gaze direction, helping to separate the voices of speakers in the environment and overcoming the cocktail party effect.

[0053] According to some embodiments of the present invention, if a user hears selectively directional sound output by a bone conduction hearing device and there is ambiguity in the sound, the system displays or projects a synchronized mouth, etc., using synchronization to an external display (e.g., mobile device 206), light generated by light emitting diodes in smart contact lenses, filtered sound, etc. According to one or more embodiments, the augmented reality object (e.g., mouth, or light) is a visual aid that helps the user understand what is being heard.

[0054] According to an embodiment of the present invention, a simulation of mouth movements or animated objects aligned with the sound (voice) that the user is hearing with a bone conduction hearing device can be controlled in playback mode. According to an embodiment of the present invention, in playback mode, the user can adjust the speed of the playback so that the animated objects and voice synchronization are played back at a comfortable speed (i.e., for the user). summary

[0055] According to one or more embodiments of the present invention, a method includes providing a smart contact lens including an array of beamforming microphones (101), where the smart contact lens is worn by a user and the array of beamforming microphones receives sounds from different directions, determining a gaze direction of the user (102), filtering the sounds received by the array of beamforming microphones based on the gaze direction of the user to determine sounds to be filtered (103), and outputting the filtered sounds via a bone conduction hearing device (104).

[0056] The methodology of embodiments of the present disclosure may be particularly well suited for use in electronic devices or alternative systems. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "processor," "circuit," "module," or "system."

[0057] Additionally, any of the methods described herein may include the additional steps of organizing a computer system and providing resource services to the computer system. Additionally, a computer program product may include a tangible computer readable recordable medium having code adapted to be executed to perform one or more of the methods described herein, and may include providing a system having separate software modules.

[0058] One or more embodiments of the present invention, or elements thereof, may be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory to perform exemplary method steps. FIG. 3 illustrates a computer system that may be useful for implementing one or more aspects or elements of the present invention, or both, and is representative of a cloud computing node according to an embodiment of the present invention. Referring now to FIG. 3, the cloud computing node 10 is merely one example of a suitable cloud computing node and is not intended to limit in any way the scope of use or functionality of the embodiments of the present invention described herein. Instead, the cloud computing node 10 may implement and / or perform any of the functionality described herein above.

[0059] Cloud computing node 10 includes computer system / server 12, which may operate with numerous other general purpose or special purpose computing system environments or configurations. Known computing systems, environments or configurations, or combinations thereof, suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments, including any of the above systems or devices.

[0060] Computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be implemented in a distributed cloud computing environment where tasks are performed by remote processing system devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

[0061] 3, computer system / server 12 in cloud computing node 10 is shown in the form of a general purpose computing device. Components of computer system / server 12 include, but are not limited to, one or more processors, or processing units 16, system memory 28, and a bus 18 connecting various system components, including system memory 28, to processor 16.

[0062] Bus 18 may represent any of several types of one or more bus structures, including a memory bus or memory controller, a peripheral bus, a graphics accelerator port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Interconnect (PCI) bus.

[0063] The computer system / server 12 typically includes a variety of computer readable media, which may be any available media that is accessible by the computer system / server 12, including both volatile and non-volatile media, removable and non-removable media.

[0064] The system memory 28 may include computer readable media such as volatile memory, such as random access memory (RAM) 30, or cache memory 32, or both. The computer server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown and typically referred to as a "hard drive"). Although not shown, a removable, non-volatile magnetic disk drive (e.g., a "floppy disk") and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such an example, each may be connected to the bus 18 by one or more media interfaces. As further shown and described below, the memory 28 may include at least one program product having a set (at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0065] The programs / utilities 40 include a set (at least one) of program modules 42, which by way of example and not limitation, may include an operating system, as well as one or more application programs, other program modules, and program data stored in memory 28. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methodologies of embodiments of the present invention as described herein.

[0066] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24; one or more devices that may allow a user to interact with the computer system / server 12, or any device that allows the computer system / server 12 to communicate with one or more computing devices (e.g., network cards, modems, etc.), or both. Such communication may occur through an input / output (I / O) interface 22. Additionally, the computer server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (the Internet), or combinations thereof, through a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 through a bus 18. It should be understood that other hardware and / or software components, not shown, could be used in combination with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, and external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0067] Thus, one or more embodiments may employ software operating on a general purpose computer or workstation. With reference to Figure 3, such an implementation may include, for example, a processor 16, memory 28, and an input / output interface 22 to a display 24, and external device(s) 14, such as a keyboard, pointing device, etc. As used herein, the term "processor" is intended to include any processing device, such as, for example, a CPU (Central Processing Unit), or other form of processing circuitry, or both. Additionally, the term "processor" may refer to one or more individual processors. The term "memory" is intended to include memory associated with a processor, or CPU, including, for example, random access memory (RAM) 30, read-only memory (ROM), fixed memory devices (e.g., hard drive 34), removable memory devices (e.g., diskettes), flash memory, and the like. Additionally, the phrase "input / output interface" as used herein is intended to contemplate, for example, an interface to one or more mechanisms for inputting data to a processing unit (e.g., a mouse), and one or more subsequent mechanisms for providing results associated with the processing unit (e.g., a printer). The processor 16, memory 28, and input / output interface 22 may be interconnected, for example, via a data processing unit 12 extending 18 as part of the data processing unit 12. For example, suitable interconnections via bus 18 may be provided to a network interface 20, such as a network card, which may provide for interfacing to a computer network, and to a media interface, such as a diskette or CD-ROM drive, which may provide for interfacing to a suitable media.

[0068] Thus, computer software containing instructions or code for carrying out the methodologies of the present invention, as described herein, can be stored in one or more associated memory devices (e.g., ROM, fixed or removable memory) and can be loaded partially or entirely into a CPU (e.g., in RAM) and implemented when ready for use. Such software can include, but is not limited to, firmware, persistent software, microcode, etc.

[0069] A data processing system suitable for storing and / or executing program code will include at least one processor 16 coupled directly or indirectly to memory elements 28 through a system bus 18. The memory elements may include local memory used during the actual implementation of the program code, bulk storage, and a cache memory 32 that provides temporary storage of at least some of the program code to reduce the number of times the code must be recalled from bulk storage during implementation.

[0070] Input / output, or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I / O controllers.

[0071] Network adapters 20 may also be coupled to the system to enable the data processing system to become coupled to other data processing systems, or remote printers, or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are some of the types of network adapters currently available.

[0072] As used herein, including the claims, a "server" includes a physical data processing system that runs a server program (eg, system 12 of FIG. 3). Such a physical server may or may not include a display and keyboard.

[0073] Any of the methods described herein may include the additional step of providing a system including distinct software modules embodied on a computer-readable storage medium; the modules may include any or all of the appropriate elements, for example, as set forth in the block diagrams and / or described herein; by way of example and not limitation, any one, some, or all of the described module blocks, or sub-modules / sub-blocks, or both, are described. The method steps may then be performed as described above using the distinct software modules, or sub-modules, or both, of the system executing on one or more hardware processors 16. Additionally, a computer program product may include a system having distinct software modules, including a computer-readable storage medium having code adapted to be implemented to perform one or more method steps described herein.

[0074] One example user interface that may be used in some cases is HyperText Markup Language (HTML) code provided by a server or the like to a browser on a user's computing device, which is parsed by the browser on the user's computing device to generate a Graphical User Interface (GUI).

[0075] The present disclosure may be a system, method, or computer program product, or a combination thereof, including a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform the features of the present disclosure.

[0076] A computer readable recording medium may be a tangible device capable of holding and storing a plurality of instructions for use by an instruction execution device. A computer readable medium may be, for example, but not limited to, an electrical recording device, a magnetic recording device, an optical recording device, an electro-magnetic recording device, a semiconductor recording device, or any suitable combination thereof. More specific examples of computer readable recording media include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a punch card, or a mechanically encoded device having protruding structures within a groove having instructions recorded therein, and any suitable combination thereof. As used herein, a computer-readable recording medium is not to be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves such as wave guides or other communications media (e.g., light pulses passing through fiber optic cable), or electrical signals communicated through wires.

[0077] The computer programs described herein can be downloaded from a computer readable recording medium into each computing / processing device, or can be downloaded to an external computer or external recording device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, and combinations thereof. The network can include copper cables, fiber optics, wireless networks, routers, firewalls, switches, gateway computers, and edge servers, and combinations thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions into a computer readable recording medium in each computing / processing device for storage.

[0078] Computer readable program instructions for performing operations of the present invention may be either source code or object code written in any combination of programming languages, including assembler instructions, Instruction Set Architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or one or more procedural programming languages, such as Smalltalk®, object-oriented programming languages ​​such as C++, the “C” programming language, or similar programming languages. The computer readable program instructions may execute entirely on the user computer, partially on the user computer as a stand-alone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user computer through any type of network, including a local area network (LAN), a wide area network (WAN), or the connection may be to an external computer (e.g., through an Internet service provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute the computer readable program instructions and use state information of the computer readable program instructions to personalize the electronic circuitry to perform features of the invention.

[0079] Features of the invention described herein have been described with reference to flowchart instructions and / or block diagrams of methods, apparatus (systems), and computer-readable recording media and computer program products according to embodiments of the invention. It will be understood that any combination of flowchart illustrations and / or block diagrams and / or blocks in flowchart illustrations and / or block diagrams can be implemented by computer-readable program instructions.

[0080] These computer readable program instructions can be provided to a general purpose computer, a special purpose computer, or other processor or other programmable data processing device to produce a machine, which when executed by the processor of the computer or other programmable data processing device produces means for implementing the functions / operations specified in the block or blocks of the flowcharts and block diagrams, or combinations thereof. These computer readable program instructions that direct the computer, programmable data processing device, and other devices, or combinations thereof, to function in a particular manner can also be stored on a computer readable recording medium, and the computer readable recording medium having instructions stored therein constitutes an article of manufacture containing instructions that implement the functional / operational features specified in the block or blocks of the flowcharts and block diagrams, or combinations thereof.

[0081] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device and cause a computer implemented process to perform a series of operational steps on the computer, other programmable apparatus, or other device to implement the functions / acts specified in the blocks of the flowcharts and block diagrams or in a combination thereof.

[0082] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and possible implementation operations of systems, methods, and computer program products according to various embodiments of the present invention. In this respect, the flowcharts or block diagrams may represent modules, segments, or portions of instructions, which include one or more executable instructions for implementing a particular logical function(s) (or functions). In some alternative implementations, the functions described in the blocks may be performed other than as illustrated. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. It is also noted that the illustration of the blocks in the block diagrams and / or the illustration of the flowcharts or combinations thereof may be implemented by a system based on special purpose hardware that performs a particular function or operation or executes specific purpose hardware and computer instructions.

[0083] The terms used herein are for the purpose of describing particular embodiments and are not intended to be limiting of the disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, when the terms "comprise", "comprising", or both are used herein, it will be understood that they specify the presence of the stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups, or combinations thereof.

[0084] Equivalents of all corresponding structures, materials, acts, and step-plus-function means in the following claims are intended to include any structure, material, or act for performing a function in combination with elements of other claims as specifically claimed. The description of the present invention has been provided for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosed form. Many modifications or variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments have been selected and described in order to best explain the principles and practical application of the invention and to enable others skilled in the art to understand the invention in various embodiments with various modifications as adapted to the specific applications envisioned.

Claims

1. 1. A computer-implemented method comprising: determining, by a processor, a direction of sound received by an array of beamforming microphones receiving sound from different directions; determining, by a processor, a focus based on the direction of the sound; outputting, by a processor, a signal based on the focus; receiving, by an audio output device, the signal output by the processor; outputting, by the audio output device, a filtered sound that is a customized processing of a certain sound received by the array of beamforming microphones based on the signal output by the processor; and providing a smart contact lens including said array of beamforming microphones; the smart contact lens is worn by a user; method.

2. providing a smart contact lens including an array of beamforming microphones, the smart contact lens being worn by a user, the array of beamforming microphones receiving sounds from different directions; determining a gaze direction of the user; filtering the sounds received by the array of beamforming microphones based on a gaze direction of the user to determine sounds to be filtered; and outputting the filtered sound through a bone conduction hearing device; A method comprising:

3. Determining the direction of the sound includes determining a line of sight of the user. The method of claim 1 , comprising:

4. filtering the sounds received by the array of beamforming microphones based on the gaze direction of the user to determine sounds to be filtered. The method of claim 3 , comprising:

5. The filtered sound is output through a bone conduction hearing device. The method according to claim 4.

6. The gaze direction of the user is determined based on a source localization of the sound received by the array of beamforming microphones.

6. The method of claim 3, 4 or 5.

7. 1. A computer-implemented method comprising: determining, by a processor, a direction of sound received by an array of beamforming microphones receiving sound from different directions; determining, by a processor, a focus based on the direction of the sound; outputting, by a processor, a signal based on the focus; receiving, by an audio output device, the signal output by the processor; outputting, by the audio output device, a filtered sound based on the signal output by the processor, the filtered sound being a customized version of the sound received by the array of beamforming microphones; and Displaying a visual simulation synchronized with said filtered sound. A method comprising:

8. Additionally, varying the playback speed of the filtered sound and the visual simulation. The method of claim 7, comprising:

9. Further comprising providing a smart contact lens including said array of beamforming microphones; the smart contact lenses are worn by a user, and the visual simulation is displayed by the smart contact lenses. The method of claim 7.

10. The visual simulation is displayed on an external display. The method of claim 7.

11. and displaying a visual enhancement synchronized with the filtered sound. The method of claim 1 , comprising:

12. and varying the playback speed of the filtered sound and the visual enhancements. The method of claim 11 , comprising:

13. 1. A non-transitory computer readable medium containing computer executable instructions that, when executed by a computer, cause the computer to perform a method of filtering sound, the method comprising: determining, by a processor, a direction of sound received by an array of beamforming microphones receiving sound from different directions; determining, by a processor, a focus based on the direction of the sound; outputting, by a processor, a signal based on the focus; receiving, by an audio output device, the signal output by the processor; outputting, by the audio output device, a filtered sound that is a customized processing of a certain sound received by the array of beamforming microphones based on the signal output by the processor; and providing a smart contact lens including said array of beamforming microphones; the smart contact lens is worn by a user; A computer-readable recording medium.

14. Determining the direction of the sound includes determining a line of sight of the user.

14. The computer readable storage medium of claim 13, comprising:

15. filtering the sounds received by the array of beamforming microphones based on the gaze direction of the user to determine sounds to be filtered.

15. The computer readable storage medium of claim 14, comprising:

16. The filtered sound is output through a bone conduction hearing device.

16. The computer readable storage medium of claim 15.

17. The gaze direction of the user is determined based on a source localization of the sound received by the array of beamforming microphones.

17. A computer readable recording medium according to claim 14, 15 or 16.

18. and displaying a visual simulation or augmentation synchronized with the filtered sound, the visual simulation or augmentation being displayed by at least one of the smart contact lenses and an external display.

14. The computer readable storage medium of claim 13.

19. Additionally, varying the playback speed of the filtered sound and the visual simulation.

20. The computer readable storage medium of claim 18, comprising:

20. 1. A system comprising: an array of beamforming microphones; a processor in electronic communication with the array of beamforming microphones and configured to determine a direction of sound received by the array of beamforming microphones, determine a focus based on the direction of sound, and output a signal based on the focus; an audio output device configured to receive the signal output by the processor and to output filtered audio, the filtered audio being configured to preferentially process certain sounds received by the beamforming microphones based on the signal output by the processor; a mobile device including the processor; wherein the array of beamforming microphones is disposed on a contact lens.

21. The contact lens is paired with at least one of the mobile device and the audio output device.

21. The system of claim 20.

22. comprising program code means adapted to carry out the method according to any one of claims 1 to 12 when run on a computer. Computer program.

Images