Enhancing Audio for Hearing-Impaired Persons in a Shared Listening Environment

Abstract

An electronic device and method are provided for enhancing audio for a user with a hearing impairment in a shared listening environment. The electronic device receives first audio content from a media source and detects a first user with a hearing impairment as a wearer of a head-mounted audio device. The electronic device modifies one or more characteristics of the first audio content based on an audio enhancement profile associated with the detected first user. Thereafter, the electronic device generates second audio content based on the modification and shares the generated second audio content with the head-mounted audio device.

Description

[0001] Cross - Reference to Related Applications / Incorporation by Reference

[0002] None. Field of the Invention

[0003] Various embodiments of the present disclosure relate to assistive technologies for disabled persons. More specifically, various embodiments of the present disclosure relate to an electronic device for enhancing audio for hearing - impaired persons in a shared listening environment. Background Art

[0004] Media devices such as televisions are generally capable of playing audio through various types of audio devices, for example, playing audio through built - in speakers, wireless speakers, or wired / wireless headphones. Many media devices generally do not allow simultaneous sound output in various configurations (e.g., internal speakers and Bluetooth , internal speakers and headphone jacks, headphone jacks and Bluetooth , or internal speakers, headphone jacks, and Bluetooth ). Thus, in a shared listening environment, if a user with a hearing impairment wears an audio device such as wireless headphones, the user may experience poor audibility of the audio output from the audio device.

[0005] In some cases, the processing of wireless audio may take more time on the audio device. As a result, a user with a hearing impairment may experience echo. In some other cases, while watching video content, the user may also experience lip - syn problems, where the audio of the video content may be delayed compared to the video content. Due to these problems, in a shared listening environment, compared to other users with normal hearing levels, the user may have a poor listening experience.

[0006] As described in the remainder of this application and with reference to the drawings, by comparing the described systems with some aspects of the present disclosure, the limitations and disadvantages of conventional and traditional methods will become apparent to those skilled in the art. Summary of the Invention

[0007] Basically as shown in at least one of the figures and / or described in combination with at least one of the figures, as more fully set forth in the claims, there is provided an electronic device and method for enhancing audio for hearing - impaired persons in a shared listening environment.

[0008] These and other features and advantages of the present disclosure will be understood from the following detailed description of the disclosure and the review of the drawings, in which like reference numerals throughout the figures indicate like parts. Brief Description of the Drawings

[0009] Figure 1It is a schematic diagram of an exemplary listening environment for users with normal hearing levels and users with hearing impairments according to an embodiment of the present disclosure.

[0010] Figure 2 It is a block diagram showing an exemplary electronic device for providing enhanced audio content for a user with a hearing impairment according to an embodiment of the present disclosure.

[0011] Figure 3 It is a diagram showing an exemplary operation for providing enhanced audio content for a user with a hearing impairment according to an embodiment of the present disclosure.

[0012] Figure 4 It is a diagram showing an exemplary scenario for enhancing audio content for a user with hearing loss according to an embodiment of the present disclosure.

[0013] Figure 5 It is a diagram showing a scenario for enhancing audio for a user with a hearing impairment based on the user's audiogram according to an embodiment of the present disclosure.

[0014] Figure 6 It is a flowchart showing an exemplary operation for providing enhanced audio content for a user with hearing loss according to an embodiment of the present disclosure. Detailed Description

[0015] Implementations described below may be found in the disclosed electronic device and method for providing enhanced audio content for a user with a hearing impairment in a shared listening environment. Exemplary aspects of the present disclosure provide an electronic device (e.g., a smart television or any media player) that can receive audio content from a media source and enhance the received audio content based on an audio enhancement profile of a user with a hearing impairment. In a shared listening environment, a traditional media device plays audio through a connected audio device, creating a shared listening experience for all users in the shared listening environment. While a user with normal hearing levels (e.g., a user with minimal hearing loss to no hearing loss) may have an enjoyable listening experience of the audio, a user with a hearing impairment may not be able to share the same experience. The reason may be attributed to the audio characteristics of the played audio, as such characteristics are typically optimized only for users with normal hearing levels. To enhance the listening experience of a user with a hearing impairment, the disclosed electronic device can receive source audio content from a media source and can detect a user with a hearing impairment as a wearer of a head-mounted audio device (e.g., wireless headphones). After such detection, the electronic device can modify certain characteristics of the source audio content (e.g., the hearing level (in dB) of certain audio frames) based on an audio enhancement profile associated with the detected first user to generate enhanced audio content that can be optimized for a user with a hearing impairment. The electronic device can share the enhanced audio content with the head-mounted audio device for playback, such that the user with a hearing impairment can have the same listening experience as a user with normal hearing levels. The enhanced audio content can improve one or more of speech intelligibility, auditory comfort, sound quality, audibility over a wide frequency range, or the perception of natural loudness of the sound associated with the first audio content.

[0016] Figure 1 is a diagram of an exemplary listening environment for users with normal hearing levels and users with hearing impairments, in accordance with an embodiment of the present invention. Refer to Figure 1, shows a listening environment 100, which includes an electronic device 102, a display device 104, a media source 106, a first user 108, a head-mounted audio device 110 for the first user 108, an image capture device 112, a second user 114, and an audio reproduction system 116 for the second user 144. The electronic device 102 can communicate with other electronic devices via a communication network 118, such as the display device 104, the media source 106, the head-mounted audio device 110, or the image capture device 112. In some embodiments, one or more devices (such as the display device 104 and the image capture device 112) can be communicatively coupled to the electronic device 102 via an input / output (I / O) port of the electronic device 102.

[0017] The listening environment 100 can be a shared listening environment, which can help to play enhanced audio content for a user with hearing impairment (such as the first user 108) without affecting the listening experience of other users (such as the second user 114) with normal hearing levels in the listening environment 100. Examples of the listening environment 100 can include, but are not limited to, a cinema, a conference hall, a concert hall, an auditorium, or a home environment where audio / video resources are shared among friends and family members.

[0018] The electronic device 102 can include suitable logic, circuitry, and interfaces, which can be configured to receive first audio content from the media source 106 and can modify certain characteristics of the first audio content based on the audio enhancement profile of the first user 108 to generate second audio content. For example, the characteristics of the first audio content can include the signal energy level or amplitude level of the audible frequency band in the audio frames of the first audio content. The second audio content can be shared with the head-mounted audio device 110, which can be worn by the first user 108.

[0019] In an embodiment, the electronic device 102 can be a portable media player, which can be configured to communicate with the display device 104 via a wired or wireless connection. Examples of such implementations of the electronic device 102 can include, but are not limited to, a digital media player (DMP), a micro console, a television tuner (such as an Advanced Television Systems Committee (ATSC) tuner), a set-top box, an over-the-top (OTT) player, a digital media streamer, a media extender / regulator, or a digital media hub.

[0020] In another embodiment, the electronic device 102 can be a media player that supports display. In such a case, all the functions of the display device 104 can be incorporated into the electronic device 102 without departing from the scope of the present disclosure. Examples of such implementations of the electronic device 102 can include but are not limited to a television (TV), a smart TV, an Internet Protocol TV (IPTV), a smart phone, a personal computer, a laptop computer, a tablet computer, a wearable electronic device, or any other media device having the ability to receive, decode, and play content from over-the-air broadcast signals or from Internet-based communication signals via a wired or satellite network.

[0021] The display device 104 can include appropriate logic, circuitry, and interfaces that can be configured to display video content that can be received from the media source 106. The display device 104 can be implemented by several known techniques, such as but not limited to a liquid crystal display (LCD) monitor, a light emitting diode (LED) monitor, a plasma monitor, or an organic LED (OLED) display technology, or other display devices. According to an embodiment, the display device 104 can refer to the display screen of a see-through display, a projection-based display, an electrochromic display, or a transparent display.

[0022] The media source 106 can include appropriate logic, circuitry, and interfaces that can be configured to transmit media content (such as audio / video content) to the electronic device 102. In an embodiment, the media source 106 can be implemented as a storage device that can be configured to store first audio content and corresponding video content related to the first audio content. Examples of such implementations of the media source 106 can include but are not limited to a pen drive, a flash USB stick, a hard disk drive (HDD), a solid state drive (SSD), and / or a Secure Digital (SD) card. In another embodiment, the media source 106 can be implemented as a media streaming server that can transmit the first audio content and the corresponding video content to the electronic device 102 via the communication network 118. In another embodiment, the media source 106 can be a television tuner (such as an ATSC tuner) that can be configured to receive a digital television (DTV) signal from an over-the-air broadcast network and extract the first audio content and the corresponding video content from the received DTV signal. Thereafter, the media source 106 can transmit the extracted first audio content and the corresponding video content to the electronic device 102.

[0023] In Figure 1 the figure, the media source 106 and the electronic device 102 are shown as two separate devices. However, the present disclosure is not limited thereto, and in some embodiments, the functions of the media source 106 can be incorporated into the electronic device 102 in whole or at least in part without departing from the scope of the present disclosure.

[0024] The head-mounted audio device 110 may include suitable logic, circuitry, and interfaces that may be configured to receive second audio content from the electronic device 102 and play the received second audio content. The second audio content may be a modified version of the first audio content and may allow a first user 108 with hearing impairment to optimally hear the second audio content. Examples of the head-mounted audio device 110 may include, but are not limited to, over-ear headphones, in-ear headphones, clip-on headphones, bone conduction headphones, hearing aids, or head-mounted wearable devices (such as smart glasses, head-mounted displays (e.g., virtual / augmented / mixed reality headsets)). In an exemplary embodiment, the head-mounted audio device 110 may be a wireless headset that may rely on wireless communication protocols such as Wi-Fi, Bluetooth or Bluetooth Low Energy (BLE) to receive the second audio content from the electronic device 102.

[0025] In an embodiment, the head-mounted audio device 110 may include a microphone for capturing the speech content of the first user 108 and an input device for receiving input from the first user 108. Examples of the input device may include, but are not limited to, a keypad, a knob, a touch screen, a touchpad, a gesture controller, or a voice-controlled input device. In another embodiment, the input device may be communicatively coupled to the head-mounted audio device 110 via a wireless network. In such an implementation, the input device may be one of the following: a smart phone, a joystick, a game controller, or any other device that may be wirelessly paired with the head-mounted audio device 110.

[0026] The image capture device 112 may include suitable logic, circuitry, and interfaces that may be configured to capture an image of the listening environment 100. For example, an image may be captured to detect the wearing state of the head-mounted audio device 110 and / or whether the first user 108 with hearing impairment is the wearer of the head-mounted audio device 110 in the detected wearing state. In an embodiment, the image capture device 112 may be communicatively coupled to the electronic device 102 via a communication network 118. In another embodiment, the image capture device 112 may be included in the electronic device 102. Examples of the image capture device 112 may include, but are not limited to, an image sensor, a wide-angle camera, an action camera, a closed-circuit television (CCTV) camera, a camcorder, a digital camera, a camera phone, and / or other image capture devices.

[0027] The audio reproduction system 116 may include suitable logic, circuitry, and interfaces that may be configured to receive first audio content (i.e., original source audio) from the electronic device 102 and play the received first audio content for a user (e.g., the second user 114) with normal hearing levels in the listening environment 100. In an embodiment, the audio reproduction system 116 may be communicatively coupled to the electronic device 102 via a wired or wireless network. In another embodiment, the audio reproduction system 116 may be an internal speaker system of the electronic device 102. For example, the audio reproduction system 116 may include a set of internal speakers, wireless speakers, smart speakers, wired speakers, woofers, subwoofers, tweeters, soundbars, loudspeakers, optical audio devices, etc. In an exemplary embodiment, the audio reproduction system 116 may correspond to a surround sound system having a specific speaker layout / configuration (e.g., 5:1 or 2:1 speaker configuration).

[0028] The communication network 118 may include a communication medium through which two or more of the electronic device 102, the display device 104, the media source 106, the head-mounted audio device 110, the image capture device 112, and the audio reproduction system 116 may communicate with each other. The communication network 118 may be a wired or wireless communication network. Examples of the communication network 118 may include, but are not limited to, the Internet, a Wireless Fidelity (Wi-Fi) network, a Personal Area Network (PAN), a Local Area Network (LAN), or a Metropolitan Area Network (MAN).

[0029] Various devices in the listening environment 100 may be configured to connect to the communication network 118 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Zig-Bee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0030] In operation, the electronic device 102 can receive first audio content from a media source 106. Thereafter, the electronic device 102 can detect a first user 108 with a hearing impairment as a wearer of the head-mounted audio device 110. In an embodiment, the electronic device 102 can control the image capture device 112 to capture an image of the listening environment 100 that may include the first user 108 and the head-mounted audio device 110. Based on the captured image, the electronic device 102 can detect the first user 108 as a wearer of the head-mounted audio device 110. For example, the electronic device 102 can locally implement a face recognition method on the image to detect the first user 108, and then use a trained deep neural network (DNN) to determine whether the detected first user 108 is a wearer of the head-mounted audio device 110.

[0031] In another embodiment, the electronic device 102 can control the head-mounted audio device 110 to capture a voice input from the wearer of the head-mounted audio device 110. Thereafter, the electronic device 102 can generate a voice profile (which may include a digital audio fingerprint) based on the voice input, and can compare the generated voice profile with a set of stored user voice profiles. Based on such comparison, the electronic device 102 can detect the first user 108 as a wearer of the head-mounted audio device 110.

[0032] After such detection, the electronic device 102 can retrieve an audio enhancement profile associated with the detected first user 108. The audio enhancement profile can include a representation of a human's hearing level (e.g., in decibels (dB)) over a wide range of different pitches or audible frequencies. For example, the audio enhancement profile can be an audiogram, which can include the hearing curves of both the left and right ears of the first user 108. Based on the retrieved audio enhancement profile, the electronic device 102 can modify one or more characteristics of the first audio content. For example, one such characteristic can be the hearing level (dB) of the audio frequency range associated with the first audio content. In the case where the audio enhancement profile indicates that the hearing level (dB) of the first user 108 for the 1000 Hz ± 250 Hz band is lower than a required threshold, the sound amplitude level of the first audio content in the 1000 Hz ± 250 Hz band can be increased to improve the hearing level. Other examples of such characteristics can include, but are not limited to, the amplitude of multiple audio frames of the first audio content, the dynamic range of the first audio content, the amplitude level of the voice frames in the first audio content, and the adjustment speed for adjusting the gain level of the audio frames of the first audio content. For example, in Figure 3 、 Figure 4 and Figure 5 further details regarding the modification of the characteristics are provided.

[0033] The electronic device 102 can generate second audio content based on a modification of first audio content. The second audio content can be an enhanced version of the first audio content and can be optimized according to the hearing impairment condition of the first user 108. Such a condition can be reflected by data points (such as hearing levels) in the audio enhancement profile of the first user 108.

[0034] Once the second audio content is generated, the electronic device 102 can share the generated second audio content with the head-mounted audio device 110. When receiving from the electronic device 102, the head-mounted audio device 110 can control the playback of the second audio content for the first user 108. In some embodiments, the electronic device 102 can control the audio reproduction system 116 to simultaneously play the first audio content when the second audio content is played on the head-mounted audio device 110 for a shared listening experience. For example, when the first user 108 hears the selectively enhanced second audio content through the head-mounted audio device 110, the second user 114 can simultaneously hear the first audio content through the audio reproduction system 116.

[0035] Without departing from the scope of the present disclosure, modifications, additions, or omissions can be made to Figure 1 For example, the listening environment 100 can include more or fewer elements than those shown and described in the present disclosure.

[0036] Figure 2 is a block diagram showing an exemplary electronic device for providing enhanced audio content for a user with a hearing impairment according to an embodiment of the present disclosure. In combination with the elements from Figure 1 the elements in Figure 2 are explained. Referring to Figure 2 , a block diagram 200 of the electronic device 102 is shown. The electronic device 102 can include a circuit 202, a memory 204, an input / output (I / O) device 206, and a network interface 208. In at least one embodiment, the electronic device 102 can further include a display device 104, an image capture device 112, and an audio reproduction system 116.

[0037] Circuit 202 may include suitable logic, circuitry, and interfaces that may be configured to execute program instructions associated with different operations to be performed by electronic device 102. Circuit 202 may include one or more dedicated processing units, which may be implemented as an integrated processor or a cluster of processors that together perform the functions of the one or more dedicated processing units. Circuit 202 may be implemented based on many processor technologies known in the art. Examples of implementations of Circuit 202 may be x86-based processors, graphics processing units (GPUs), reduced instruction set computing (RISC) processors, application specific integrated circuit (ASIC) processors, complex instruction set computing (CISC) processors, microcontrollers, central processing units (CPUs), and / or other computing circuitry.

[0038] Memory 204 may include suitable logic, circuitry, and interfaces that may be configured to store program instructions to be executed by Circuit 202. In at least one embodiment, Memory 204 may store files, such as first audio content, second audio content, and video content. Memory 204 may also store user information that may be associated with the listening environment 100 and may be accessible to a user of electronic device 102. For example, the user information may include voice profiles, facial recognition information, or audio enhancement profiles of first user 108 and second user 114. Examples of implementations of Memory 204 may include, but are not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), hard disk drive (HDD), solid state drive (SSD), CPU cache, and / or secure digital (SD) card.

[0039] I / O device 206 may include suitable logic, circuitry, and interfaces that may be configured to receive input and provide output based on the received input. I / O device 206, which includes various input and output devices, may be configured to communicate with Circuit 202. Examples of I / O device 206 may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, a microphone, a display device (such as display device 104), or a button.

[0040] Network interface 208 may include suitable logic, circuitry, and interfaces that may be configured to facilitate communication between Circuit 202, display device 104, media source 106, image capture device 112, and / or audio reproduction system 116 via communication network 118. Network interface 208 may be implemented by using various known techniques to support wired or wireless communication of electronic device 102 with communication network 118. Network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuit.

[0041] The network interface 208 can be configured to communicate with a network via wireless communication, such as the Internet, an intranet, or a wireless network, such as a cellular telephone network, a wireless local area network (LAN), and a metropolitan area network (MAN). The wireless communication can use one or more of a plurality of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), protocols for e-mail, instant messaging, and Short Message Service (SMS).

[0042] As Figure 1 described in the functions or operations performed by the electronic device 102 can be performed by the circuit 202. For example, in Figure 3 , Figure 4 , Figure 5 and Figure 6 the operations performed by the circuit 202 are described in detail.

[0043] Figure 3 is a diagram showing exemplary operations for providing enhanced audio content for a user with hearing impairment according to an embodiment of the present disclosure. In combination with elements from Figure 1 and Figure 2 is explained Figure 3 . Referring to Figure 3 , a block diagram 300 is shown, which shows exemplary operations from 302 to 310 as described herein. The exemplary operations shown in the block diagram 300 can start at 302 and can be performed by any computing system, device, or apparatus, such as by Figure 1 or Figure 2 of the electronic device 102. Although shown using discrete boxes, the exemplary operations associated with one or more boxes of the block diagram 300 can be divided into additional boxes, combined into fewer boxes, or eliminated, depending on how the exemplary operations are implemented.

[0044] At 302, media content 302A can be received. Circuit 202 can receive media content 302A from media source 106. The received media content 302A can include first audio content and / or video content associated with the first audio content. As an example, media content 302A can be a television program that can include video and audio associated with the video. As another example, media content 302A can be a radio show or podcast that can include only audio.

[0045] At 304, wearer detection can be performed. Circuit 202 can detect a first user 108 with hearing impairment as a wearer of the head-mounted audio device 110. Various embodiments for detecting the first user 108 as a wearer of the head-mounted audio device 110 are described herein.

[0046] In an embodiment, circuit 202 can control the image capture device 112 to capture an image of the listening environment 100 that can include the first user 108 and the head-mounted audio device 110. Based on the captured image, circuit 202 can detect the first user 108 as a wearer of the head-mounted audio device 110. For example, circuit 202 can locally implement a face recognition method on the image to detect the first user 108, and then use a trained DNN to determine whether the detected first user 108 is a wearer of the head-mounted audio device 110.

[0047] In another embodiment, circuit 202 can control the head-mounted audio device 110 to capture a voice input from the wearer of the head-mounted audio device 110, and circuit 202 can generate a voice profile (which can include a digital audio fingerprint) based on the voice input. Thereafter, circuit 202 can compare the generated voice profile with a set of stored user voice profiles, and based on such comparison, circuit 202 can detect the first user 108 as a wearer of the head-mounted audio device 110.

[0048] In another embodiment, circuit 202 can control the display device 104 to present a user interface (UI) that includes a list of registered user profiles as user-selectable options. Through an input device associated with the head-mounted audio device 110, circuit 202 can receive a selection of the registered user profile associated with the first user 108. The registered user profile can include information associated with the hearing impairment of the first user 108. Based on such selection, the wearer of the head-mounted audio device 110 can be detected as the first user 108.

[0049] At 306, an audio enhancement profile associated with the detected first user 108 can be retrieved from the memory 204. The audio enhancement profile can include a representation of a human's hearing level (in decibels (dB)) over a wide range of pitches or audible frequencies. For example, the audio enhancement profile can be represented by an audiogram, which can include the hearing curves for both the left and right ears of the first user 108. Figure 5 An example of an audiogram as an audio enhancement profile is provided.

[0050] In the case where the audio enhancement profile for the detected first user 108 is not available, the audio enhancement profile can be created as described herein. The circuit 202 can control the head-mounted audio device 110 to sequentially play a set of test tones at a corresponding set of audible frequencies. When a corresponding test tone is played on the head-mounted audio device 110, the loudness of each test tone can be increased. Doing so can allow the detected first user 108 of the head-mounted audio device 110 to determine the loudness level of the corresponding test tone that the first user 108 can clearly hear. When the set of test tones is sequentially played on the head-mounted audio device 110, the circuit 202 can receive a set of user inputs. Each user input in the set of user inputs can indicate the hearing threshold level (in dB) of the corresponding test tone. Based on the received set of user inputs, the circuit 202 can generate a hearing curve on the audiogram as the audio enhancement profile of the detected first user 108.

[0051] At 308, audio enhancement can be performed for the detected first user 108. For such enhancement, the circuit 202 can modify one or more characteristics of the first audio content based on the audio enhancement profile associated with the detected first user 108 (obtained at 302C). Examples of such characteristics can include, but are not limited to, the amplitude of multiple audio frames of the first audio content, the amplitude level of the audible frequency band of the first audio content, the dynamic range of the first audio content, the amplitude level of the speech frames in the first audio content, and the adjustment speed of the gain level for adjusting the audio frames of the first audio content.

[0052] In one case, the first user 108 may have high-frequency hearing loss (typically 2000 Hz or higher). As a result, the first user 108 may have difficulty understanding the speech of women, children, the songs of birds, or other high-pitched sounds. Alternatively, the first user 108 may have low-frequency hearing loss. As a result, the first user 108 may have difficulty hearing conversations within a crowd or in a noisy environment and where there is background noise, or hearing the bass of music. In such a case, the circuit 202 may determine the type of hearing loss associated with the detected first user 108 based on an audio enhancement profile. Based on the determined type of hearing loss, the circuit 202 may determine the audible frequency band in which the hearing level of the detected first user 108 is below a threshold hearing level (in dB). Thereafter, the circuit 202 may selectively amplify the amplitude levels of a plurality of audio frames of the first audio content including the determined audible frequency band. The selective amplification may correspond to a modification of a first characteristic of the first audio content.

[0053] In another case, it may be detected that the first user 108 has mild hearing loss, which may narrow the hearing range to 60 dB to 70 dB and may reduce the dynamic range of the hearing of the first user 108. In contrast, a user with a normal hearing level may have a hearing range of approximately 100 dB between the threshold and the uncomfortable loudness level. Herein, the dynamic range may be defined as the ratio of the strongest or loudest part of an audio to the weakest or softest part of the audio. In such a case, the circuit 202 may determine a first dynamic range of the first audio content and a second dynamic range of the hearing of the detected first user 108 based on an enhancement profile. Thereafter, the circuit 202 may use a compressor, an expander, or a noise gate to modify the first dynamic range of the first audio content such that the difference between the modified first dynamic range and the second dynamic range is minimized. In other words, the dynamic range of the audio content may be modified such that it is suitable for the dynamic range of the hearing of the first user 108. The modification of the first dynamic range may correspond to a modification of a second characteristic of the first audio content.

[0054] In another case, the first user 108 may have severe to profound hearing loss, where the cochlear high-frequency region may be severely damaged such that it may not be possible to restore the audibility of audio samples (which contain high-frequency signals) through conventional amplification. In such a case, audibility may be achieved by converting the high-frequency signal components of the first audio content into low-frequency signal components. The circuit 202 may extract the high-frequency signal components of the first audio content and may convert the extracted high-frequency signal components into low-frequency signal components. The conversion of the extracted high-frequency signal components may correspond to a modification of a third characteristic of the first audio content.

[0055] Circuit 202 may generate second audio content based on a modification of the first audio content. The second audio content may be a modified form of the first audio content and may be adapted to address deficiencies associated with the hearing impairment of the first user 108. Specifically, the second audio content may improve one or more of the speech clarity, auditory comfort, sound quality, audibility over a wide frequency range, or perceived natural loudness of the sounds associated with the first audio content.

[0056] At 310, the audio content may be shared with the head-mounted audio device 110. Circuit 202 may share the generated second audio content with the head-mounted audio device 110 via a wireless network (such as Bluetooth or Wi-Fi). In an embodiment, circuit 202 may control the audio reproduction system 116 to play the first audio content simultaneously with the playback of the second audio content on the head-mounted audio device 110 for a shared listening experience. Thus, when the first user 108 listens to the second audio content through the head-mounted audio device 110, the second user 114 may simultaneously listen to the first audio content through the audio reproduction system 116. Since both the first audio content and the second audio content are played simultaneously, both users (the first user 108 and the second user 114) may be able to enjoy a shared listening experience in a common listening environment.

[0057] Figure 4 is a diagram illustrating an exemplary scenario for enhancing audio content for a user with a hearing impairment in accordance with an embodiment of the present disclosure. In combination with elements from Figure 1 、 Figure 2 and Figure 3 to explain Figure 4 . Referring to Figure 4 , an exemplary scenario 400 is shown. In the exemplary scenario 400, a television 402 and a display panel 404 of the television, a first user 406, a head-mounted audio device 408, an image capture device 410, a second user 412, and an audio reproduction system 414 are shown. Herein, the television 402, the display panel 404, the head-mounted audio device 408, the image capture device 410, and the audio reproduction system 414 may be exemplary implementations of the electronic device 102, the display device 104, the head-mounted audio device 110, the image capture device 112, and the audio reproduction system 116, respectively.

[0058] The television 402 can receive media content through a media source 106 (such as an ATSC tuner). The media content can include first audio content and video content associated with the first audio content. The television 402 can control the display panel 404 to display the video content associated with the first audio content. For a first user 406 with hearing impairment, the television 402 can generate second audio content (as described in Figure 3 ) based on the modification of one or more characteristics of the first audio content. The second audio content can be a modified form of the first audio content and can be adapted to address the hearing defect of the first user 406.

[0059] Generally, when the television 402 wirelessly shares the second audio content with the head-mounted audio device 408, compared with playing back the video content on the display panel 404, the playback of the second audio content may be delayed (e.g., dozens to hundreds of milliseconds). This may be attributed to the wireless audio processing method for encoding or decoding the second audio content and / or the wireless sharing method for the second audio content. In the case of video playback, the processing method for the video content may cause a delay that may not be visually perceptible to the human eye during the display of the video content, especially compared with the delay associated with the playback of the second audio content. This may cause a lip-sync problem for the first user 406. In addition, in the case where the audio reproduction system 414 is wired to the television 402 and plays the first audio content, compared with the delay associated with the playback of the second audio content, the delay associated with the playback of the first audio content can be negligible. In such a case, the sound of the first audio content may be mixed with the sound of the second audio content, causing the first user 406 to hear an echo.

[0060] To alleviate such problems, the circuit 202 can calculate a first processing delay associated with the encoding of the second audio content for wireless sharing with the head-mounted audio device 408. The circuit 202 can also calculate a second processing delay associated with the decoding of the second audio content at the head-mounted audio device 408. Thereafter, the circuit 202 can calibrate the display time of the video content on the display panel 404 based on the calculated first processing delay and second processing delay. For example, such calibration can include offsetting the display time of the video content by a time equivalent to the sum of the calculated first processing delay and second processing delay. Such calibration can synchronize the playback of the video content with the playback of the second audio content on the head-mounted audio device. In addition, the circuit 202 can also calibrate the playback time of the first audio content through the audio reproduction system 414 to synchronize with the playback of the video content 304B on the display panel 404.

[0061] Figure 5is a diagram showing a scenario for enhancing audio for a user with a hearing impairment based on the user's audiogram. In conjunction with elements from Figure 1 , Figure 2 , Figure 3 and Figure 4 to explain Figure 5 . Referring to Figure 5 , a first audiogram 500 and a second audiogram 504 are shown. The first audiogram 500 includes a hearing curve 502 of a first user 108 with a hearing impairment, and the second audiogram 504 includes a hearing curve 506 of another user with normal hearing levels over a wide audible frequency range.

[0062] In the first audiogram 500, for the audible frequency band from 0 Hertz (i.e., Hz) to 250 Hz, the hearing curve 502 of the first user 108 indicates normal hearing levels. However, for audible frequency bands above 250 Hz, the hearing curve 502 of the first user 108 indicates a decline in hearing levels, especially for audible frequencies above 2000 Hz. The first audiogram 500 can be referred to as an audio enhancement profile of the first user 108. To correct the hearing levels of the first user 108, the circuit 202 can compare the hearing curve 502 of the first user 108 with the hearing curve 506 of another user with normal hearing levels. Based on this comparison, the circuit 202 can determine one or more characteristics of the first audio content to be modified to generate a second audio content. For example, in this case, the circuit 202 can determine a plurality of audible frequency bands, where the signal energy levels of the plurality of audible frequency bands may have to be selectively amplified to calibrate the hearing levels (in dB) of the first user 108. Thereafter, the circuit 202 can determine a plurality of audio frames of the first audio content as a characteristic (as one of the determined one or more characteristics) based on the determination that the determined plurality of audio frames include the determined plurality of audible frequency bands. Thereafter, the circuit 202 can selectively amplify the determined plurality of audio frames of the first audio content to generate a second audio content. The second audio content can be shared with the head-mounted audio device 110 for playback.

[0063] In the case where the first user 108 has a hearing impairment in the left ear or the right ear, the circuit 202 can select a first audio portion as one of the left-channel audio or the right-channel audio of the first audio content based on the determination that the hearing curve 502 of the first user 108 indicates the hearing impairment condition of one of the left ear or the right ear of the first user 108. The circuit 202 can generate a second audio content based on the modification of one or more characteristics (such as hearing levels (in dB)) of the selected first audio portion of the first audio content.

[0064] Figure 6is a flowchart showing exemplary operations for providing enhanced audio content for a user with a hearing impairment. In conjunction with elements from Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 to explain Figure 6 . Referring to Figure 6 , flowchart 600 is shown. The operations from 602 to 610 can be implemented by any computing system, such as by the electronic device 102 of Figure 1 or Figure 2 . The operation can start at 602 and proceed to 604.

[0065] At 602, first audio content can be received from the media source 106. In at least one embodiment, the circuit 202 can be configured to receive the first audio content from the media source 106.

[0066] At 604, a first user 108 with a hearing impairment can be detected as a wearer of the head-mounted audio device 110. In at least one embodiment, the circuit 202 can be configured to detect the first user 108 with a hearing impairment as a wearer of the head-mounted audio device 110.

[0067] At 606, one or more characteristics of the first audio content can be modified based on an audio enhancement profile associated with the detected first user 108. In at least one embodiment, the circuit 202 can be configured to modify one or more characteristics of the first audio content based on the audio enhancement profile 606 associated with the detected first user 108.

[0068] At 608, second audio content can be generated based on the modification. In at least one embodiment, the circuit 202 can be configured to generate the second audio content based on the modification.

[0069] At 610, the generated second audio content can be shared with the head-mounted audio device 110. In at least one embodiment, the circuit 202 can be configured to share the generated second audio content with the head-mounted audio device 110. Control can pass to the end.

[0070] Although flowchart 600 is shown as discrete operations, such as 602, 604, 606, 608, and 610. However, in certain embodiments, such discrete operations can be further divided into additional operations, combined into fewer operations, or eliminated, depending on the implementation, without departing from the essence of the disclosed embodiments.

[0071] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium having instructions stored thereon that are executable by a machine and / or computer to operate an electronic device, such as electronic device 102. The instructions may cause the machine and / or computer to perform operations that include: receiving first audio content from a media source 106 and detecting a first user 108 with a hearing impairment as a wearer of a head-mounted audio device 110. The operations may further include: modifying one or more characteristics of the first audio content based on an audio enhancement profile associated with the detected first user 108. The operations may further include: generating second audio content based on the modification and sharing the generated second audio content with the head-mounted audio device 110.

[0072] Exemplary aspects of the present disclosure may include an electronic device (such as Figure 1 electronic device 102) that includes circuitry (such as circuitry 202) that may be communicatively coupled to a head-mounted audio device (such as Figure 1 head-mounted audio device 110). The circuitry may be configured to receive first audio content from a media source (such as Figure 1 media source 106) and detect a first user (such as first user 104) with a hearing impairment as a wearer of the head-mounted audio device. The circuitry may further be configured to modify one or more characteristics of the first audio content based on an audio enhancement profile associated with the detected first user. By way of example and not limitation, one or more characteristics may include the amplitude of multiple audio frames of the first audio content, the amplitude level of the audible frequency band of the first audio content, the dynamic range of the first audio content, the amplitude level of speech frames in the first audio content, the adjustment speed of a gain level for adjusting audio frames of the first audio content, and the like. Thereafter, the circuitry may be configured to generate second audio content based on the modification and share the generated second audio content with the head-mounted audio device.

[0073] According to an embodiment, the circuitry may further be configured to control an audio reproduction system (such as Figure 1 audio reproduction system 114) to play the first audio content concurrently with the playback of the second audio content on the head-mounted audio device for a shared listening experience. For example, the audio reproduction system may include one or more of the following: an external wireless speaker, a set of internal speakers, an external wired speaker, a sub-woofer, a tweeter, a soundbar, or an optical audio device.

[0074] According to an embodiment, an electronic device may include an image capture device (e.g., image capture device 112) communicatively coupled to a circuit. The circuit may be configured to control the image capture device to capture an image of a listening environment (e.g., listening environment 100) including a first user and a head-mounted audio device, and detect the first user as a wearer of the head-mounted audio device based on the captured image.

[0075] According to an embodiment, an audio enhancement profile may be an audiogram that may include the hearing curves of a first user for both the left ear and the right ear of the first user. The circuit may be configured to compare the hearing curves of the first user with a reference hearing curve of a second user having a normal hearing level, and determine one or more characteristics of a first audio content for modification based on the comparison.

[0076] According to an embodiment, wherein the circuit may be further configured to select a first audio portion as one of a left-channel audio or a right-channel audio of the first audio content based on a determination that the hearing curve of the first user indicates a hearing impairment condition of one of the left ear or the right ear of the first user. Thereafter, the circuit may modify one or more characteristics of the selected first audio portion of the first audio content, and generate a second audio content based on the modification.

[0077] According to an embodiment, the circuit may be configured to control the head-mounted audio device to sequentially play a set of test tones in a corresponding set of audible frequencies. When a corresponding test tone is played on the head-mounted audio device, the loudness of each test tone in the set of test tones may be increased. Thereafter, when the set of test tones is sequentially played on the head-mounted audio device, the circuit may receive a set of user inputs. Each user input may indicate the hearing threshold level of a corresponding test tone. Based on the received set of user inputs, the circuit may generate a hearing curve on the audiogram as the audio enhancement profile of the first user.

[0078] According to an embodiment, an electronic device may include a display device (e.g., display device 104) configured to display video content that may be associated with the first audio content. The circuit may be configured to calculate a first processing delay associated with the encoding of a second audio content for wireless sharing with the head-mounted audio device, and calculate a second processing delay associated with the decoding of the second audio content at the head-mounted audio device. Thereafter, the circuit may be further configured to calibrate the display time of the video content based on the calculated first processing delay and the second processing delay to synchronize with the playback of the second audio content on the head-mounted audio device.

[0079] The present disclosure can be implemented in hardware or in a combination of hardware and software. The present disclosure can be implemented in a centralized manner in at least one computer system or in a distributed manner in which different elements are distributed in several interconnected computer systems. A computer system or other device suitable for performing the methods described herein may be appropriate. The combination of hardware and software may be a general-purpose computer system with a computer program that, when loaded and executed, can control the computer system so that the computer system performs the methods described herein. The present disclosure can be implemented in hardware that includes a portion of an integrated circuit that also performs other functions.

[0080] The present disclosure can also be embedded in a computer program product that includes all the features enabling the implementation of the methods described herein and, when loaded in a computer system, is capable of executing these methods. In this context, a computer program refers to any expression of a set of instructions represented in any language, code, or notation, intended to cause a system with information processing capabilities to perform a specific function either directly or after any one or both of the following: a) conversion to another language, code, or notation; b) reproduction in a different material form.

[0081] Although the present disclosure has been described with reference to certain embodiments, those skilled in the art will understand that various changes can be made and equivalents can be substituted without departing from the scope of the present disclosure. In addition, many modifications can be made to adapt a situation or material to the teachings of the present disclosure without departing from its scope. Therefore, the present disclosure is not limited to the disclosed embodiments, but the present disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. An electronic device, comprising: a circuit communicatively coupled to a head-mounted audio device, wherein the circuit is configured to: receive first audio content from a media source; detect a first user with a hearing impairment as a wearer of the head-mounted audio device; modify one or more characteristics of the first audio content based on an audio enhancement profile associated with the detected first user; generate second audio content based on the modification; and share the generated second audio content with the head-mounted audio device, wherein the circuit is further configured to: calculate a first processing delay associated with encoding of the second audio content for wireless sharing with the head-mounted audio device; calculate a second processing delay associated with decoding of the second audio content at the head-mounted audio device; and calibrate a display time of video content associated with the first audio content based on the calculated first processing delay and the second processing delay to synchronize with playback of the second audio content on the head-mounted audio device.

2. The electronic device according to claim 1, wherein, the circuit is configured to control an audio reproduction system to play the first audio content simultaneously with playback of the second audio content on the head-mounted audio device for a shared listening experience.

3. The electronic device according to claim 2, wherein, The audio reproduction system includes one or more of the following: an external wireless speaker, a set of internal speakers, an external wired speaker, a subwoofer, a tweeter, a soundbar, or an optical audio device.

4. The electronic device according to claim 1, further comprising an image capture device communicatively coupled to the circuit, wherein, The circuit is further configured to: control the image capture device to capture an image of a listening environment including the first user and the head-mounted audio device; and detect the first user as the wearer of the head-mounted audio device based on the captured image.

5. The electronic device according to claim 1, wherein, The audio enhancement profile is an audiogram that includes a hearing curve for both the left ear and the right ear of the first user.

6. The electronic device according to claim 5, wherein, The circuit is further configured to: compare the hearing curve of the first user with a reference hearing curve of a second user with normal hearing levels; and determine the one or more characteristics of the first audio content for the modification based on the comparison.

7. The electronic device according to claim 5, wherein, The circuit is further configured to: select a first audio portion as one of the left-channel audio or the right-channel audio of the first audio content based on a determination that the hearing curve of the first user indicates a hearing impairment condition of one of the left ear or the right ear of the first user; modify the one or more characteristics of the selected first audio portion of the first audio content; and generate second audio content based on the modification.

8. The electronic device according to claim 1, wherein, The circuit is further configured to: control the head-mounted audio device to sequentially play a set of test tones in a corresponding set of audible frequencies, wherein when a corresponding test tone is played on the head-mounted audio device, the loudness of each test tone in the set of test tones increases; receive a set of user inputs when the set of test tones is sequentially played on the head-mounted audio device, wherein each user input indicates a hearing threshold level of a corresponding test tone; and generate a hearing curve on the audiogram as the audio enhancement profile of the first user based on the received set of user inputs.

9. The electronic device according to claim 1 further includes a display device configured to display video content associated with the first audio content.

10. The electronic device according to claim 1, wherein, The one or more characteristics include the amplitude of multiple audio frames of the first audio content, the amplitude level of the audible frequency band of the first audio content, the dynamic range of the first audio content, the amplitude level of the speech frames in the first audio content, or the adjustment speed for adjusting the gain level of the audio frames of the first audio content.

11. A method performed by an electronic device, comprising: Receiving first audio content from a media source; Detecting a first user with hearing impairment as a wearer of a head-mounted audio device; Modifying one or more characteristics of the first audio content based on an audio enhancement profile associated with the detected first user; Generating second audio content based on the modification; And Sharing the generated second audio content with the head-mounted audio device, wherein the method further includes: Calculating a first processing delay associated with the encoding of the second audio content for wireless sharing with the head-mounted audio device; Calculating a second processing delay associated with the decoding of the second audio content at the head-mounted audio device; and Calibrating the display time of the video content associated with the first audio content based on the calculated first processing delay and second processing delay to synchronize with the playback of the second audio content on the head-mounted audio device.

12. The method according to claim 11 further comprises: Controlling an audio reproduction system to play the first audio content simultaneously with the playback of the second audio content on the head-mounted audio device for a shared listening experience.

13. The method according to claim 11 further includes: Controlling an image capture device to capture an image of the listening environment including the first user and the head-mounted audio device; And Detecting the first user as the wearer of the head-mounted audio device based on the captured image.

14. The method according to claim 11, wherein, The audio enhancement profile is an audiogram that includes the hearing curves of the first user for both the left ear and the right ear of the first user.

15. The method according to claim 14 further includes: Comparing the hearing curve of the first user with a reference hearing curve of a second user with normal hearing level; And Determining the one or more characteristics of the first audio content for the modification based on the comparison.

16. The method according to claim 14 further includes: Selecting a first audio portion as one of the left-channel audio or the right-channel audio of the first audio content based on a determination that the hearing curve of the first user indicates a hearing impairment condition in one of the left ear or the right ear of the first user; Modifying the one or more characteristics of the selected first audio portion of the first audio content; And Generating second audio content based on the modification.

17. The method according to claim 11 further includes: Controlling the head-mounted audio device to sequentially play a set of test tones in a corresponding set of audible frequencies, wherein when a corresponding test tone is played on the head-mounted audio device, the loudness of each test tone increases; When the set of test tones is sequentially played on the head-mounted audio device, receive a set of user inputs, where each user input indicates the hearing threshold level of the corresponding test tone; and Based on the received set of user inputs, generate a hearing curve on a hearing diagram as an audio enhancement profile for a first user.

18. A non-transitory computer-readable medium having computer-executable instructions stored thereon, the computer-executable instructions, when executed by a computer in an electronic device, cause the electronic device to perform operations, the operations including: Receive first audio content from a media source; Detect a first user with a hearing impairment as a wearer of a head-mounted audio device; Modify one or more characteristics of the first audio content based on an audio enhancement profile associated with the detected first user; Generate second audio content based on the modification; And Share the generated second audio content with the head-mounted audio device, where the operations further include: Calculate a first processing delay associated with the encoding of the second audio content for wireless sharing with the head-mounted audio device; Calculate a second processing delay associated with the decoding of the second audio content at the head-mounted audio device; and Calibrate the display time of video content associated with the first audio content based on the calculated first processing delay and second processing delay to synchronize with the playback of the second audio content on the head-mounted audio device.

Images