Audio processing using hearing loss data

Abstract

A method of processing audio, the method comprising: receiving audio data for playback on a playback device, the audio data comprising a target audio component; receiving masked audio data; obtaining hearing loss data associated with a listener; obtaining back discharge level data; calculating a target excitation pattern based on the target audio component and the return discharge level data; calculating a masked excitation pattern based on the masked audio data; calculating a signal gain based on the target excitation pattern, the masked excitation pattern, and the hearing loss data; and applying the signal gain to the target audio component to produce a compensated target audio component.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 502,391, filed May 15, 2023, and U.S. Provisional Application No. 63 / 588,909, filed October 9, 2023, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] This disclosure generally relates to signal processing, and more specifically to techniques, apparatus, and computer-executable instructions for processing signals related to multimedia content. In particular, this disclosure relates to compensation for hearing impairment. Background Technology

[0003] Recent statistics show that approximately 15% of American adults report hearing impairment, with age being the most significant predictor of hearing loss. Men are almost twice as likely as women to experience hearing loss. About 2% of adults aged 45 to 54 have disabling hearing loss, rising to 8.5% in the 55 to 64 age group and 25% in the 65 to 74 age group. Additionally, it is estimated that approximately 28 million American adults could benefit from using hearing aids.

[0004] For decades, the hearing aid industry has been developing hearing aids and associated fitting rules to provide solutions for people with hearing loss. The goal of these fitting rules is to describe a hearing optimization algorithm (or hearing aid) that specifies what gain should be applied at what frequencies to reproduce the signal captured by the microphone on the hearing aid. Early on, around 1950, the half-gain rule was a popular method for determining frequency-dependent gain. In the absence of dynamic range processing capabilities, the idea was to apply a gain equivalent to half the hearing loss (in decibels (dB)). Therefore, this fitting rule provided a single gain target for any particular type of hearing loss, applicable to all input levels.

[0005] In the 1990s, compression-based fitting rules emerged, among which the DSL (Desired Perceptual Rank) and NAL-NL1 (National Acoustics Labs, Nonlinear, Version 1) were particularly popular. The idea behind NAL fitting rules is to attempt to equalize loudness relationships within speech frequencies, rather than normalize them. Specifically, it is assumed that speech intelligibility is maximized if all speech frequencies are amplified so that they are heard at equal loudness. In other words, NAL methods do not preserve loudness relationships between speech frequencies; instead, they aim to maximize the information about speech provided to the listener given a particular degree of hearing loss. In NAL terminology, this is described as maximizing 'effective audibility,' rather than 'audibility' itself or a perceptual rank. Summary of the Invention

[0006] The perception of externalized loudness, timbre, spectral naturalness, spatial audio, and speech intelligibility depends on a variety of contextual factors, such as the degree of hearing loss, auditory masking from media components (e.g., music and effects), media playback levels, and ambient sounds like background noise. These factors are not simply additive but interact in complex ways. The preservation of media artistry, naturalness, surround sound, immersion, realism, and speech intelligibility in media reproduced by audible devices (e.g., headphones, earbuds, hearing aids, etc.) can greatly benefit from algorithms that optimize for the combined negative impact of hearing loss, the effects of playback levels, and the degree of masking from other media components and ambient sounds. This applies to content types beyond speech (including traditional linear media content such as music and film) as well as non-linearly generated or interactive media content such as AR, VR, and games.

[0007] This paper describes a technique that preserves the media artistry (e.g., loudness, timbre, spectral naturalness, externalization of spatial audio and speech intelligibility, surround sound, immersion, realism, etc.) of media reproduced through playback devices (e.g., headphones, earplugs, hearing aids, AR / VR devices, head-mounted displays, etc.) for individuals with hearing impairments. The described technique is compatible with MPEG-I-enabled devices and systems.

[0008] The first aspect relates to a method for processing audio for a listener on a playback device, the method comprising: receiving audio data for playback on the playback device, the audio data including a target audio component; receiving masked audio data; obtaining hearing loss data associated with the listener; obtaining playback level data; calculating a target excitation mode based on the target audio component and the playback level data; calculating a masked excitation mode based on the masked audio data; calculating a signal gain based on the target excitation mode, the masked excitation mode, and the hearing loss data; and applying the signal gain to the target audio component to produce a compensated target audio component.

[0009] Playback level data describes the (frequency-dependent) conversion of the digital input signal to the sound pressure level reaching the listener's ear. Playback level data can be, for example, the actual playback level data received from the playback device. Alternatively, the playback level data can be estimated based on knowledge of the playback device type, or default playback level data can be used.

[0010] The masked audio data may include background audio components contained in the audio data, and the masked excitation mode may include a background excitation mode based on the background audio components and playback level data.

[0011] The masking audio data may also include ambient sound picked up by a microphone near the listener, and the masking excitation pattern may include an ambient excitation pattern based on the ambient sound. The ambient excitation pattern may also be based on "playback device data" describing the effect of the playback device on the ambient sound. This playback device data may include, for example, the transfer function of a set of active filters for a wearable device.

[0012] On the other hand, a method for audio rendering is provided, the method comprising: receiving an audio bitstream including a target audio component and a background audio component; receiving audio scene information, the audio scene information including a listener's current head pose and / or gaze; processing the audio scene information to obtain an audio scene state; rendering audio data based on the scene state to obtain a rendered target audio component and a rendered background audio component; receiving hearing loss data associated with the listener; receiving playback level data; calculating a target excitation mode based on the target audio component and the playback level data; calculating a masking excitation mode based on the background audio component and the playback level data; calculating a signal gain based on the target excitation mode, the masking excitation mode, and the hearing loss data; and applying the signal gain to the rendered target audio component to produce a compensated rendered target audio component.

[0013] Another aspect relates to an audio renderer comprising: a decoder for receiving and decoding an audio bitstream to provide input audio including a target audio component and a background audio component; a scene controller configured to receive audio scene information and provide an audio scene state based on the audio scene information; and a rendering pipeline configured to: render audio data based on the scene state to obtain a rendered target audio component and a rendered background audio component; receive hearing loss data associated with a listener; receive playback level data; calculate a target excitation mode based on the target audio component and the playback level data; calculate a masking excitation mode based on the background audio component and the playback level data; calculate a signal gain based on the target excitation mode, the masking excitation mode, and the hearing loss data; and apply the signal gain to the rendered target audio component to produce a compensated rendered target audio component.

[0014] This audio rendering method is suitable for virtual reality (augmented reality, extended reality) applications with accessible interfaces. For users with hearing impairments, calculating and applying signal gain according to this disclosure may be advantageous.

[0015] This audio rendering may include receiving descriptive data that indicates whether a specific audio stream of audio data is a target audio component or a background audio component. This descriptive data may exist in existing or future VR / AR / XR audio rendering devices (e.g., rendering devices compliant with the MPEG-I standard).

[0016] The proposed method enables combinatorial optimization by utilizing one or more of the following: • Understand ambient sounds, for example, via a microphone attached to earbuds or other wearable devices; • Understand the acoustic media playback levels, for example, given the current volume control settings on a playback device, understand the media digital signal level and its corresponding acoustic sound pressure level; • With adequate speech intelligibility, media speech can be separated from media background audio (if applicable). • Understand the listener's hearing loss (e.g., their audiogram, or age + gender to obtain the group average audiogram). • To determine the perceived ambient sound level, we need to understand the frequency-dependent attenuation of the acoustic environment while the listener is wearing a hearing-enabled device, and also the operating mode of these hearing-enabled devices (e.g., passive mode, pass-through mode, or acoustic noise reduction mode).

[0017] Given some or all of the information above, the perceptual loudness model can predict the optimal media playback gain as a function of frequency and time for joint optimization of the aforementioned contextual, interactive, and personalized factors.

[0018] Executable instructions for performing these functions may optionally be included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0019] This technology used to perform these functions can complement or replace other methods used to perform similar functions. Attached Figure Description

[0020] To better understand the various embodiments described, reference should be made to the following embodiments in conjunction with the accompanying drawings, wherein similar descriptions refer to corresponding parts throughout the drawings.

[0021] Figure 1 The illustration shows spectral naturalness and timbre distortion caused by hearing impairment according to some embodiments.

[0022] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0023] Figure 3 yes Figure 2 A more detailed block diagram of media audio processing in the diagram.

[0024] Figure 4 An MPEG-I audio renderer implementing an embodiment of the present invention is shown.

[0025] Figure 5 The system architecture of the MPEG-I renderer is shown.

[0026] Figure 6 The processing in the MPEG-I renderer is shown.

[0027] Figure 7 It shows Figure 6 The rendering stage in the rendering pipeline.

[0028] Figure 8 This is a flowchart illustrating a method for improving the user experience of hearing-impaired individuals according to some embodiments. Detailed Implementation

[0029] The following description illustrates exemplary methods, parameters, devices, etc. However, it should be understood that such description is not intended to limit the scope of this disclosure, but is provided as a description of exemplary embodiments.

[0030] There is a need for electronic devices that improve the user experience for people with hearing impairments. For example, there is a need for electronic devices that provide users with audio that preserves artistic intent (e.g., externalization of loudness, timbre, spectral naturalness, spatial audio and speech intelligibility, surround sound, immersion, realism, etc.).

[0031] Although the following description uses terms such as "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various embodiments described, first data may be referred to as second data, and similarly, second data may be referred to as first data. Both first data and second data are data, but they are not the same data.

[0032] The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated enumerated items. It will be further understood that, when used in this specification, the terms “comprising,” “including,” “including,” and / or “comprising” indicate the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0033] Depending on the context, the term "if" may optionally be interpreted as meaning "when...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrases "if determining..." or "if [the stated condition or event] is detected" may optionally be interpreted as meaning "after determining...", "in response to determining...", "after [the stated condition or event] is detected", or "in response to detecting [the stated condition or event]".

[0034] Now focus on embodiments of the disclosed techniques that can be implemented on electronic devices. In some embodiments, the electronic device includes memory (which optionally includes one or more computer-readable storage media) and one or more processing units (CPU, GPU, DSP, ASIC, etc.). In some embodiments, the electronic device belongs to a type of device commonly used to provide AR / VR content. For example, in some embodiments, the electronic device is a smartphone, tablet, or other computing system (e.g., desktop computer, laptop computer) that includes audio and video capture capabilities (hardware capture devices locally or communicatively coupled to the device).

[0035] In some embodiments, the electronic device is implemented as part of a distributed system comprising one or more components located in different locations. For example, in some embodiments, the electronic device includes local hardware components and one or more remote hardware components (e.g., cloud-based components hosted on public and / or private cloud infrastructure, SaaS, PaaS, etc.). In some embodiments, the electronic device is or includes a smartphone, tablet, or other computing system (e.g., a desktop computer, laptop computer) with audio and video capture capabilities (hardware capture devices locally or communicatively coupled to the device).

[0036] It should be understood that the above-described device is merely one example of an electronic device, and the device may optionally have more or fewer components than described, may optionally combine two or more components, or may optionally have different component configurations or arrangements. The various components of the device described above and throughout this document are implemented in hardware, software, or a combination of both, including one or more signal processing and / or application-specific integrated circuits (ASICs).

[0037] Now let’s turn our attention to implementations on, for example, electronic devices (such as those described above).

[0038] In most cases, hearing impairment is frequency- and level-dependent, and therefore loudness loss (or recruitment) is also frequency- and level-dependent. Consequently, spectral naturalness and timbre are distorted due to hearing impairment. Figure 1 Examples of the input audio signal spectrum perceived by a listener with normal hearing (solid line 11) and the audio signal spectrum perceived by a listener with high-frequency hearing impairment (dashed line 12) are shown. Region 13 identifies loudness loss and perceived timbre variation. The hearing optimization algorithm should aim to reduce loudness loss and timbre variation by amplifying the affected signal frequencies and creating a perceived input signal spectrum (dashed line 14) that more closely approximates the spectrum perceived by a listener with non-sensory sensorineural hearing impairment.

[0039] It is important to note that the actual signal level is not the only factor. Frequency-dependent playback level data (which may include the playback frequency response of the wearable device) also plays a significant role. This playback level data allows for the conversion from the media digital signal level to the acoustic sound pressure level. Therefore, if such data is available, it is encouraged to consider it when estimating hearing optimization gains. It should be noted that these gains are applied to the input signal (input audio stream) to obtain the final output signal.

[0040] The combined effects of hearing impairment and noise masking when listening to the media Hearing impairment and noise masking (both stemming from environmental and media context) can significantly impact how media is perceived, including: • Difficulty in understanding dialogue. Important details of speech may be lost or reduced due to more obvious masking of media background or ambient sounds, higher hearing thresholds, and modifications to the perception of loudness as a function of signal level, resulting in decreased dialogue intelligibility.

[0041] • Perceived timbre distortion or lack of spectral naturalness. Since both hearing loss and auditory masking are frequency- and level-dependent processes, the amount of information lost due to hearing impairment or ambient noise masking is also frequency-dependent, leading to a degraded perceived timbre and spectral naturalness. For example, in low-frequency traffic noise, a listener may not be able to hear the bass lines of a music track, while other instruments and vocals may be loud enough. In this case, increasing the playback volume will not solve the problem of timbre distortion.

[0042] • Loss of detail in instruments or audio objects and background sounds. Quiet elements of instruments (such as high-frequency overtones, subtle effects of string instruments, singer's breathing, and soft background sounds) may be inaudible or reduced in loudness due to hearing loss and noise masking.

[0043] • The perception of dynamics and dynamic range is altered. It is well known that hearing impairment can lead to changes in dynamic perception, as well as the relationship between signal level and perceived loudness.

[0044] • Reduced spatial awareness and externalization of binaural audio. Externalization and spatial awareness are governed by subtle cues such as a small amount of acoustic environment simulation. In the presence of noise masking and hearing loss, these important cues may be lost or perceived at a reduced level, thus negatively impacting the spatial audio experience.

[0045] Challenges in applying hearing aid fitting rules to media playback Given the substantial achievements in optimizing speech intelligibility for hearing aids in noisy environments, and considering the impact of environmental noise and hearing loss on media consumption as outlined in previous sections, applying hearing aid fitting rules and algorithms to media playback in noisy environments could be considered. However, in practice, this often results in suboptimal or even poor performance due to various reasons. For example: • According to traditional fitting rules or hearing aid algorithms, signal components that are not considered to be within the speech frequency range are considered unimportant. Therefore, signal components below a few hundred hertz are not adequately amplified compared to the degree of hearing loss or masking at these frequencies; similar problems occur at higher frequencies. Consequently, traditional hearing aid fitting rules can lead to significant losses in the perception of bass and treble frequencies in media types where these frequency ranges are important, such as music and movies.

[0046] • Optimizing speech intelligibility by equalizing the loudness of speech components can lead to significant distortion of the timbre and spectral naturalness of speech content, and more importantly, it can also cause distortion of the timbre of non-speech content (such as music or movies).

[0047] Traditional hearing aid fitting rules often amplify speech components, boosting them above the estimated ambient noise masking threshold. In noisy environments, this results in a very small dynamic range for the processed speech, as aggressive processing is unsuitable for other media content such as music or movies.

[0048] • It is well known that source separation of stereo media content (e.g., for separating speech from other media content) is difficult in hearing aids because hearing aids operate largely independently, and there are very few devices available for cross-device communication (e.g., communication between the left and right hearing aids and between the hearing aid and the media playback device). However, in media playback use cases, the media is strictly separated from ambient sound, so no source separation algorithm is needed to separate speech from the acoustic environment.

[0049] • Hearing aids do not have any concept of two separate masking sources (i.e., signals from the environment and background sounds from the media). It is important to consider both, as both can have considerable masking effects, and / or some of these content components may be considered important and require optimization, depending on the use case.

[0050] • Without further information about volume settings, wearable device sensitivity, and frequency response, it may be impossible to determine the exact media playback level, parameters that are crucial for environmental noise compensation during media playback.

[0051] General Implementation This disclosure presents an integration method / technology designed to enable listeners to hear content as intended by the content creator, meaning that hearing media content is like: • No significant hearing loss, and / or • There is no environmental noise and / or media background masking, and / or • The playback level is close to the reference playback level, or at least provides a tone close to that reproduced at the playback reference level.

[0052] The techniques described below include a perceptual model framework for listening to and optimizing media in noisy environments, and a system for simultaneously optimizing audio for environmental masking and hearing loss. These techniques improve upon the shortcomings described above regarding existing fitting and compensation techniques (e.g., as described in the introduction above).

[0053] Figure 2 A high-level system diagram according to some embodiments of the proposed method / technology is illustrated. The media playback device 20 here includes a media player 21 and a set of wearable devices 22 (e.g., a single transducer or a pair of transducers). The media player 21 can be any type of audio signal generator, such as a smartphone, home entertainment system, virtual reality system, etc. The wearable devices 22 can be a head-mounted display, head-mounted device, or earpiece worn by a listener. The wearable devices 22 include at least one earpiece (two in the illustrated case), each earpiece having a transducer (not shown), most typically a speaker. Each earpiece may also include at least one microphone 24 configured to receive ambient sound. In some examples, the wearable devices 22 may consist of a single earpiece. Furthermore, the media playback device can be a single device including both the media player 21 and the wearable devices 22, such as a head-mounted display.

[0054] Media player 21 provides an input media signal 15, which is sent to wearable device 22 via media audio processing module 23. Media player 21 can be connected to audio processing module 23 and / or wearable device 22 via wired or wireless connection. Media audio processing module 23 can be integrated into media player 21 or wearable device 22, or it can be a separate entity configured to intercept signals from media player 21, process the signals, and send the signals to wearable device 22. Such a separate entity can be part of a cloud processing solution. Sound captured by wearable device 22 can be transmitted back to media player 21 and / or media audio processing module 23.

[0055] The media audio processing module 23 is configured to receive ambient audio data 25, which is obtained from one or more microphones 24 on the wearable device 22, or from any other sound detection device.

[0056] The media audio processing module 23 is further configured to receive acoustic isolation data (or alternatively referred to as playback device data 26) associated with the wearable device 22. Playback device data 26 may include a passive portion relating to the masking effect of the wearable device 22 on external sounds (e.g., data describing sound attenuation due to the wearable device physically blocking the ear canal). This passive portion will be associated with a specific type and design of the wearable device 22 and may be obtained directly from the wearable device (e.g., during setup). Playback device data 26 may further include an active portion relating to the transfer function of an active filter in the wearable device 22. Such a filter may attenuate or amplify sounds at different frequencies. The active portion of acoustic isolation data 26 may further depend on the selected mode of the wearable device 22, such as active noise cancellation (ANC) mode or pass-through mode. Therefore, the active portion may be dynamic and influenced by user input. The current operating mode of the wearable device 22 may be an explicit portion of playback device data 26.

[0057] The media audio processing module 23 is further configured to obtain playback level data 27 associated with the wearable device 22. This data describes the (frequency-dependent) conversion of the digital input signal 15 to the actual sound pressure level generated by the speakers of the wearable device 22. This data will depend on the specific type and design of the wearable device 22 and involves the frequency characteristics of the speakers in the wearable device 22. In some embodiments, the actual playback data is received from a playback device, but if this is not possible, the playback level data can be estimated, for example, based on the type of playback device used, or a default value can be used.

[0058] The media audio processing module 23 is further configured to obtain hearing loss data 28 describing the listener's hearing impairment. Hearing loss data 28 may be in the form of an audiogram or a set of parameters that allow for the estimation of the audiogram. For example, the audiogram may be obtained from a database 29 based on demographic information about the listener (e.g., age or sex at birth). Alternatively, the audiogram may be obtained by performing a hearing test or received from a hearing loss data service provider.

[0059] The media audio processing module 23 can be further configured to receive loudness adjustment data 30, such as volume control user settings.

[0060] The media audio processing module 23 is configured to process the input media 15 in response to ambient audio data 25, acoustic isolation data / playback device data 26, playback level data 27, and hearing loss data 29 to produce processed media 16. Finally, the processed media 16 is reproduced on the hearing-wearable device 22.

[0061] Implementation examples using the perceived loudness model The general approach outlined above will now be discussed further regarding specific implementations based on the perceptual loudness model, including modeling the human ear as an outer ear and middle ear transfer function H. OME and the auditory filter transfer function H ERB .

[0062] Calculation of specific loudness Perceived sound loudness can be described using excitation and specific loudness models, for example, such as... https: / / www.nidcd.nih.gov / health / statistics / quick-statistics-hearing This is discussed in the section. Such models typically begin by calculating the excitation level in a set of frequency bands, and then calculating the loudness per band based on that excitation level. This loudness per band is called the specific loudness. For simplicity and without loss of generality, the process for a single frequency band will be described, assuming that in practice, a series of spectral bands will be available, which operate more or less independently and have their own set of parameters.

[0063] Media target input signal represented in the frequency domain excitation level It is given by the following formula: (Equation 1) in, For the transfer functions of the outer and middle ear, For frequency The central auditory filter transfer function. For frequency, the unit is Hz, and This represents playback level data 27, allowing conversion from media digital signal levels to acoustic sound pressure levels, which can include the playback frequency response of the wearable device. The playback level data required to convert digital signal levels to acoustic sound pressure levels. This can depend on various factors, such as the volume control settings of the playback device. (Function) and It can be given by an appropriate model of the auditory system, and examples can be found in various textbooks on the subject.

[0064] Now you can specify a loudness (For example, signals in the band of interest) The associated loudness is calculated as follows: (Equation 2) Among them, model parameters and It may depend on the degree of hearing loss. The model constant α is a value between zero and one, preferably between 0.1 and 0.4, and will be assumed to be 0.25 in this disclosure without any bias.

[0065] Target input signal It can be equivalent to or a part of input medium 15. In the latter case, some processing is required to separate the target signal from the input signal. This will be briefly discussed below.

[0066] Loudness model calibration Model parameters C and A0 (representing 0 dB hearing loss) can be calibrated as follows. By definition, for a listener with normal hearing, a 1 kHz pitch at 40 dB SPL (40 squares) has a loudness of 1 sono: (Equation 3) Let represent the squared amplitude transfer function of the outer and middle ear. The second data point used for calibration is the 1 kHz pitch at 2 dBSPL (2 squared), which has a loudness of 0.003 sonos for a listener with normal hearing (e.g., hearing loss (HL) = 0 dB). Therefore, for a frequency of 1 kHz, we will obtain: (Equation 4) Assumption It can be approximated : (Equation 5) Similarly, assuming It can be approximated by the following formula. :

[0067] And therefore: (Equation 6) Typical value at 1 kHz = 0.79, = 0.25, and therefore , .

[0068] To obtain sensitivity that reflects cross-frequency ranges and / or differences due to hearing loss. The value can be Adjust to different values This reflects the change in signal level required to achieve the same loudness (0.003 sung) as the two-way signal. : (Equation 7) Calculation of specific loudness Incentive Model The loudness of a target signal can be greatly affected by the presence of other masking signals. In this context, the loudness of the target signal is referred to as partial loudness. Two types of masking signals can be distinguished, and excitation patterns can be calculated for both types of masking signals.

[0069] The first type is the acoustic environment in which the listener is located. If the acoustic environment has a given... The corresponding ambient sound excitation mode can be calculated using the following formula based on the signal spectrum. : (Equation 8) here, This represents the passive or active attenuation frequency response caused by the wearable device 22 on the ear or inside the ear canal (e.g., acoustic isolation data / playback device data 26). In the case of the active frequency response caused by an active filter, this can depend on the operating mode of the wearable device 22. For example, if acoustic noise cancellation (ANC) is active, more attenuation will occur at low frequencies due to ANC operation.

[0070] The second type involves components in the input media signal 15 that are considered to have a masking contribution, such as background music and effects. If these components have an equal... The corresponding excitation mode can be calculated using the following formula based on the signal spectrum. : (Equation 9) In one embodiment, a source separation method is used to, for example, employ an artificial intelligence-based algorithm to separate the media target signal. and media background signals It is separated from the input signal 15.

[0071] In the presence of masking signals and and their respective incentive models and and adjustment of hearing sensitivity (or loss) In this case, the target signal loudness Then it is given by the following formula: (Equation 10) optimization A potential optimization criterion for compensating for masking and hearing loss is the target signal. Applying time- and frequency-dependent gain, such that its predicted partial loudness is equal to its partial loudness in the absence of hearing loss and masking effects. If the signal gain is expressed as... Then, a specific loudness in a given context (e.g., with hearing loss and masking effects) is given by the following formula: (Equation 11) In the absence of context, for example, in the absence of masking and / or hearing loss contributions, the predicted loudness is determined by Equation 2, where, A = A 0: (Equation 12) Then you can make a request. To calculate Value: (Equation 13) In some implementations, the loudness adjustment data 30 may include a user-controllable loudness adjustment factor λ. In this case, the predicted loudness will be multiplied by the factor λ: (Equation 14) in this case, The value will become: (Equation 15) It should be noted that signal gain can be calculated and applied across two or more frequency bands. Furthermore, signal gain can be calculated and applied in a time-varying manner.

[0072] Figure 3 The diagram provides a schematic overview of the above method, showing the processing in the media audio processing module 23 in more detail. As described above, the media audio processing module 23 receives acoustic isolation data / playback device data 26 (including operating mode information of the wearable device), playback level data 27, ambient audio data 25, hearing loss data 28, optional loudness adjustment data 30, and input media data 15, and generates a processed media signal 16 as output.

[0073] Playback device data 26 is received by attenuation response calculation block 31, which determines the attenuation response. .

[0074] Hearing loss data 28 is received by parameter calculation block 32, which calculates parameters. .

[0075] In the illustrated example, the media input audio 15 is received by the source separation block 33 and separated into the media target signal. and media background signals (If desired). The input audio may include a set of audio components (stream or object), in which case the media target signal includes a set of target components and the media background signal includes a set of background components. The source separation block can be configured to perform separation using an appropriately trained neural network or based on characteristics of the audio components / objects (e.g., position, orientation, etc.). Alternatively, block 33 is concatenated to receive some type of descriptive data indicating, for example, whether a particular audio component is a target component or a background component, or whether the audio component is related to speech content or other content. In yet another embodiment, information about the head pose and / or gaze used can be used to distinguish between target audio components and background audio components.

[0076] Excitation calculation blocks 34, 35, and 36 respectively receive ambient audio 25 and optional media background signal. and media target signals And calculate the excitation modes respectively. , and The optimized gain is calculated based on the various excitation modes and optional loudness adjustment data, and then applied to the target media signal. If a background media signal is available, this signal is mixed with the (amplified) target signal to produce the processed output media signal.

[0077] Incentive Model , and and parameters Received by gain calculation block 37, which calculates the signal gain according to the above equation. Gain Gain application block 38 is applied to the target signal .

[0078] In the illustrated example where the input audio has been separated into target and background, the background signal is summed at point 39. It is then added back to the target signal after gain adjustment to provide the final output audio 16.

[0079] Imagined use cases Hearing loss compensation in quiet conditions In this use case, the goal is to compensate for hearing loss while assuming the listener is in a relatively quiet environment. An example could be a listener listening to music or a podcast, which could be reproduced in a personalized way by taking into account the listener's hearing loss. The masking effect of ambient audio is considered minimal, or can be completely disabled to reduce processing power and thus extend battery life—in cases where some of this processing is running on battery-powered devices. The processing device could provide means for the listener to input loudness adjustment data, allowing the perceived loudness to be adjusted in a more natural way than simply adjusting the media playback level.

[0080] Personalized media optimization in noisy environments In noisy environments, media reproduction can be improved and personalized by compensating for the listener's hearing loss and the masking effect of the acoustic environment. Assuming the listener is wearing a wearable device, such as one with headphone or earbud shape factors, the acoustic isolation or attenuation produced by the device (e.g., acoustic isolation data) is considered during optimization. Furthermore, if the wearable device has a so-called ANC or pass-through mode, the acoustic attenuation response is adjusted accordingly to represent the combined effect of placing the device in or around the ear, and its influence on ambient audio entering the ear canal. Finally, the user can adjust the playback loudness by inputting loudness adjustment data.

[0081] Personalized, context-aware dialogue enhancement Dialogue within media content may be masked by background noise and / or ambient audio. To overcome this problem, the proposed solution separates the dialogue (the target audio signal) from the background (the background audio). The target audio signal is then optimized to compensate for one or more of the following factors: (1) the listener's hearing loss; (2) the masking effect of the background audio; and (3) ambient audio (if any). By taking hearing loss into account, the dialogue enhancement features thus become personalized and context-aware.

[0082] Personalized, context-aware augmented reality One of the challenges of augmented reality is loudness management; the perceived loudness of virtual (augmented) content needs to be high enough to be heard in noisy environments, but not so high as to cause discomfort. By associating augmented reality signals with target signals and taking into account ambient audio and hearing loss, the loudness of augmented reality components can be managed entirely automatically without user interaction.

[0083] Targeted, interactive, personalized, context-aware object enhancement In some use cases, it is desirable to improve the audibility of audio components by their expected orientation or location within the listener's field of view. In such use cases, the target media signal and background signal can be separated by determining whether the target media signal is within the field of view. One embodiment of this process would be to combine object-based audio coordinates with head tracking to determine which objects are within the field of view. The field-of-view signal components are then optimized based on the assumption that all objects outside the field of view are considered background audio components. Head tracking of the listener allows the listener to indicate the objects that need optimization simply by looking at them. Therefore, this enhancement is directional, interactive, context-aware, and personalized.

[0084] In some use cases, and compared to previous examples, objects outside the field of view can be enhanced so that they remain clearly audible when masking occurs due to background media and / or ambient audio. This enhancement is particularly useful for warning signals occurring outside the field of view.

[0085] Implementation methods in VR applications Specific use cases involve VR (AR, XR, etc.) applications where head tracking data is used to render audio to simulate the effect of movement within an audio environment. Such applications sometimes feature an "accessibility user interface" that allows users to provide input designed to eliminate or at least reduce the impact of various barriers, such as hearing impairments.

[0086] The accessibility user interface can provide acoustic scene adjustments (such as adjustments to early reflections, reverberation, and distance acoustics) and directional focusing effects designed to attenuate interfering sounds from directions outside the area of ​​interest. All of these effects are generated in response to the user's 6 DoF (six degrees of freedom) posture and / or gaze in the VR / AR / XR world.

[0087] The drawback of this approach is that users must selectively and independently tune each effect within the overall effect to achieve the desired overall result. A certain level of understanding of the acoustics may also be required to correctly adjust the parameters associated with specific effects. This may be undesirable for most users with hearing impairments. By implementing the techniques discussed above, the user experience for all users, regardless of whether they have been diagnosed with hearing loss, can be improved.

[0088] Hearing optimization gain can be estimated based on, for example, audiograms, signal level information, and playback level information (e.g., as discussed above). G tIn virtual reality applications, when a user moves and explores a virtual space with 6 degrees of freedom (6 DoF), acoustic effects such as distance attenuation, occlusion, early reflections, reverberation, and diffraction are introduced into the input signal. This can affect the overall signal level.

[0089] Foreground / Target Signal Content ( X t Acoustic effects can be defined as the original input signal content or a combination of input content and the resulting acoustic effects. In some cases, it may be necessary to maintain the input signal content as a foreground / target signal, but at the same time, reduce the acoustic effects to improve intelligibility. Alternatively, acoustic effects themselves can be classified as background signals.

[0090] For example, background music and other background signals ( X m It can also be provided as an input audio stream, as part of the content creator's intent. Several foreground / target signals and background signals can be specified. Classifying the audio stream as a foreground or background signal can potentially improve the calculation of hearing-optimized gain. The gain is adjusted based on the signal levels of both the background and target signals to ensure that a hearing-impaired person can hear the target signal in the presence of background.

[0091] Alternatively, the audio stream may already be the foreground / target signal itself, mixed with a background signal. In augmented reality applications, this could be an audio signal from a real sound source mixed with background noise or ambient sound. In this case, source separation or denoising may need to be performed beforehand. An additional audio stream containing the extracted background noise can be added as input to an audio renderer (e.g., an MPEG-I audio renderer). This stream may have the same orientation as the audio stream it is mixed with. In some cases, it may be necessary to provide the “denoised” audio stream itself, or the renderer can perform denoising if “denoising” filter coefficients (e.g., frequency domain Wiener filtering) are provided.

[0092] Figure 4 An audio renderer 41 (e.g., an MPEG-I audio renderer) is shown that takes a set of audio streams 42 and 6 DoF head pose and / or gaze 43 as input and provides rendered audio 44 as output. Here, the audio renderer 41 is configured with an audiometry-optimized rendering stage 45, which renders the audiogram 46 and playback level. P(f) 47 is used as input to apply signal gain according to this disclosure. The hearing optimization rendering stage can be implemented, for example, as follows: Figure 3 The media audio processing module 23 shown.

[0093] Based on the above examples, the following parameters may be useful in systems implementing the techniques disclosed herein (e.g., MPEG-I audio renderers): Playback level is the frequency-dependent absolute playback level in dB SPL. P(f) (For example, default: 70 dBSPL) or a scalar or array of relative playback levels in dB (for example, default: 90 dB). This is in Figure 3 The middle mark is 27.

[0094] An array of frequency-related hearing loss levels in dB (e.g., default: derived from demographic data such as age and gender). This is in Figure 3 The middle mark is 28.

[0095] The target signal is used to indicate whether the audio stream is considered a foreground / target signal [true] or a background signal [false] (e.g., default: true). This will be used as descriptive data in... Figure 3 Used in box 33.

[0096] Note the playback level. P(f) This can come from the playback device (such as current volume settings and sensitivity information like the headphone's frequency response) or from a separate bitstream (e.g., an MPEG-I bitstream) (if provided as, for example, a suggested or required playback level). If no playback level data is available, the system can use common or appropriate default playback level data. For example, the system can assume that a digital signal level with a linear amplitude of 1.0 re at full digital scale corresponds to 90 dB SPL in the acoustic domain.

[0097] An advantage of applying this disclosure to a renderer (e.g., audio renderer 41) that has access to various audio sources / streams 42 is that it allows content creators to categorize certain audio sources as, for example, foreground or background sources. This can be done, for example, by utilizing parameters that indicate the appropriate priority ordering (e.g., not to "cull") for the corresponding audio sources.

[0098] For example, in future versions of MPEG-I, the "noCulling" parameter attached to each audio stream is expected to be specified as a parameter in the MPEG-I parameters. If this parameter is set to "1" or "true", it indicates that the audio stream will never be culled, which can be used to specify a target signal flag. Setting the parameter to "1" can be interpreted as indicating that the audio stream is a foreground or target signal.

[0099] In the absence of information to identify the audio stream as either "target" or "background," the input audio can be segmented into these types based on the current head posture and / or gaze. For example, if the user is looking in a certain direction, this is likely where his / her focus is, and the audio from that direction is likely the "target" audio.

[0100] Future versions of MPEG-I are also expected to include a loudness user interface as an interface for applying loudness-based audio culling. The aim is to allow existing MPEG-D / H loudness metadata to be fed into the MPEG-I renderer to perform audio culling based on absolute and relative loudness standards. This loudness information can be specified by the content creator. It can be considered as additional input during the rendering stage. Figure 4 The middle mark is 48.

[0101] In addition, additional processing blocks (e.g., a "hearing optimization rendering stage") are provided within the system implementing the techniques disclosed herein. For example, as Figure 4 The illustrated MPEG-I implementation includes a processing block for implementing optimized rendering. This rendering stage is responsible for adjusting the time-frequency correlated hearing optimization gain frame-by-frame in real time based on the input parameters described above.

[0102] Future versions of MPEG-I are expected to include spectral compensation for any user-specified "accessibility equalization". This spectral compensation will be designed to synthesize user-specified filters (in local configuration parameters within the renderer framework) and subsequently perform filtering on the binaural audio output within the "binaural spatializer" stage. It can handle both ipsilateral and heterolateral spectral compensation simultaneously. This processing will provide the opportunity to implement the hearing-optimized rendering stage 45.

[0103] Figure 5 An overview of the rendering process, including an example of an audio renderer 41, is shown. This audio renderer is implemented herein as an MPEG-I renderer. The MPEG-I renderer 41 typically operates at a global sampling frequency of 48 kHz. Figure 5 The illustration shows how the renderer 41 in this example is connected to the MPEG-H 3DA encoded audio element bitstream 51 via the MPEG-H 3DA decoder 52.

[0104] All audio elements input to the renderer (e.g., channels, objects, HOAs) have corresponding counterparts in the MPEG-I immersive audio standard, known as source types: - The object source has VR / AR specific attributes - The sound source is played back in the virtual world through virtual speaker settings. - HOA sources can be rendered into the virtual world in two different ways: either rendered independently with three degrees of freedom (user-oriented), or rendered in groups with six degrees of freedom. For all three paradigms, the encoded waveform can be directly carried from MPEG-H 3D audio to MPEG-I immersive audio without any re-encoding and without any associated quality loss.

[0105] The decoded audio is rendered along with the MPEG-I bitstream 53. The MPEG-I bitstream 53 carries an audio scene description and other metadata used by the renderer 41. The renderer 41 also has interfaces for accessing consumption environment information (e.g., LSDF) 54, scene updates during playback 55, and user position and interaction information 56. Inputs 54, 55, and 56 may be referred to as audio scene information.

[0106] The renderer 41 allows for real-time audibility of complex 6 DoF audio scenes, where users can directly interact with entities within the scene. To achieve this, the multi-threaded software architecture is divided into several workflows and components. Figure 6 The diagram shows a block diagram with all renderer components. Renderer 41 supports rendering of both VR and AR scenes. In the case of VR and AR scenes, rendering metadata and audio scene information are obtained from bitstream 53. In the case of AR scenes, listening space information is obtained during playback as an LSDF (Listener Space Description Format) file 55. The components in the block diagram are briefly described below.

[0107] The control workflow 61 serves as the entry point for the renderer 41 and is responsible for interfacing with external systems and components. Its main functions are embedded in the scene controller component 62, which coordinates the state of all entities in the 6 DoF scene and implements the interactive interface of the renderer 41. The scene controller 62 supports external updates to the modifiable properties of scene objects and can read and parse the LSDF file 55 to complete the information in the bitstream. The scene controller 62 also tracks time- or location-related properties of scene objects (e.g., interpolated position or listener proximity conditions).

[0108] Scene state 63 always reflects the current state of all scene objects, including audio elements, transforms / anchors, and geometry. Other components of the renderer can subscribe to changes in scene state 63. Before rendering begins, all objects in the entire scene are created, and their metadata is updated to reflect the desired scene configuration at the start of playback.

[0109] The stream manager 64 provides a unified interface to the renderer components to access the audio stream 42 associated with the audio elements in scene state 63. The audio stream 42 is input to the renderer 41 as a (decoded) PCM floating-point sample. The source of the audio stream 42 can be, for example, a decoded MPEG-H audio stream 51 or locally captured audio.

[0110] Clock 65 provides an interface for the renderer components to obtain the current scene time in seconds. Clock input can be, for example, a synchronization signal from another subsystem or the renderer's internal wall clock. The scene's clock input is independent of audio synchronization.

[0111] The rendering workflow 66 generates a PCM floating-point audio output signal 44. This rendering workflow is separate from the control workflow 61, and only the scene state 63 (used to transmit any changes in the 6 DoF scene) and the stream manager 64 (used to provide the input audio stream) are accessible from the rendering workflow 66 to communicate between the two workflows.

[0112] The renderer pipeline 67 makes the input audio stream 42 provided by the stream manager audible based on the current scene state. Rendering is organized in a sequential pipeline, allowing each rendering stage to implement independent perceptual effects and utilize the processing of previous and subsequent stages.

[0113] Spatializer 68 terminates renderer pipeline 67 and makes the output of the renderer stage audible as a single output audio stream suitable for the desired playback method (e.g., binaural or adaptive speaker rendering). Finally, limiter 69 provides clipping protection for the audible output signal.

[0114] Figure 7 The diagram illustrates the renderer pipeline 67, where each box represents a separate rendering stage. Rendering stages are instantiated during renderer initialization. The rendering stages are computed in the order shown in the diagram.

[0115] Due to the hearing optimization rendering stage 45 (see...) Figure 4 Benefiting from knowledge of individual source classifications (e.g., via the noCulling flag), the hearing-optimized rendering stage 45 can be implemented within the renderer pipeline 67, just before the (binaural or loudspeaker) spatializer 68. For example, the hearing-optimized rendering stage is at the end of the renderer pipeline, at... Figure 7 This occurs after the MP-HOA rendering stage. Note that during this stage, all 6 DoF acoustic effects (occlusion, diffraction, reflection, reverberation, etc.) in VR / AR / XR applications are provided as render items to be fed into the spatializer 68.

[0116] Figure 8 This is a flowchart illustrating a method for optimizing the user experience for people with hearing impairments on an electronic device, according to some embodiments. Method 300 is performed at an electronic device (e.g., an electronic device as described herein).

[0117] The electronic device receives (e.g., 302) audio data. In some embodiments, the audio data includes one or more audio streams. In some embodiments, the audio data includes loudness data. The electronic device obtains (e.g., 304) hearing loss data associated with a listener. In some embodiments, the hearing loss data is audiogram data associated with the listener. In some embodiments, the audiogram data is obtained from a hearing test associated with the listener. In some embodiments, the audiogram data is obtained from demographic information associated with the listener.

[0118] The electronic device receives (e.g., 306) contextual data including playback level data and calculates (e.g., 308) gain data (e.g., time-frequency related hearing-optimized gain) based on the audio data, hearing loss data, and contextual data. In some embodiments, the playback level data is a scalar or array of frequency-related absolute playback levels in dB SPL. In some embodiments, the playback level data is a scalar or array of relative playback levels in dB.

[0119] Electronic devices utilize gain data processing (e.g., 310) to process at least a portion of audio data to produce compensated audio data.

[0120] In some embodiments, contextual data includes signal description data. In some embodiments, the signal description data indicates whether the corresponding audio data corresponds to a foreground signal or a background signal. In some embodiments, the signal description data indicates whether the foreground signal is a speech signal or a non-speech signal.

[0121] In some embodiments that include signal description data, the electronic device determines whether the value associated with the signal description data is a first value or a second value (e.g., true or false), and based on the determination that the value associated with the signal description data is a first value (e.g., true), the gain data is a first set of gain data, and based on the determination that the value associated with the signal description data is a second value (e.g., false), the gain is a second set of gain data that is different from the first set of gain data.

[0122] For illustrative purposes, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the technology and its practical application. This enables others skilled in the art to best utilize these technologies and various embodiments with various modifications according to the specific intended use.

[0123] Although this disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. These changes and modifications should be understood to be included within the scope of this disclosure and examples as defined by the claims.

[0124] Various aspects and implementations of this disclosure can also be understood from the following enumerated exemplary embodiments (EEEs), which are not claims.

[0125] EEE1. A method for processing audio for a listener on a playback device, the method comprising: Receive audio data; Receive hearing loss data associated with the listener; Receive contextual data including one or more of the following: ambient sound data, acoustic isolation data, and playback level data; and Audio level data is calculated based on at least a portion of the audio data; Gain data is calculated based on the audio level data and the context data; and At least a portion of the audio data is processed using the gain data to produce compensated audio data.

[0126] EEE2. The method as described in EEE1, wherein the audio data includes a background component and a target component; and wherein processing at least a portion of the audio signal includes processing the target component using the gain data and abandoning processing the background component using the gain data.

[0127] EEE3. The method as described in EEE2 further includes: Before receiving the audio data, the target component of the audio data is determined at least in part based on the direction or location of one or more objects or sources represented in the audio data.

[0128] EEE4. The method of any one of EEE1 to EEE3, further comprising: Calculate one or more sensed excitation levels; and The calculation of audio level data is performed in part based on at least one of the one or more perceptual excitation levels.

[0129] EEE5. The method as described in EEE4, wherein calculating the one or more perceived excitation levels includes calculating an excitation loss function based on the hearing loss data.

[0130] EEE6. The method of any one of EEE4 to EEE5, wherein the context data includes ambient sound data, and wherein calculating the one or more perceived excitation levels includes calculating an excitation level associated with the ambient sound data.

[0131] EEE7. The method of any one of EEE4 to EEE6, wherein the context data includes acoustic isolation data, and wherein the calculation of the excitation level associated with the ambient sound data is performed in part based on the acoustic isolation data.

[0132] EEE6. The method of any one of EEE4 to EEE7 as subordinate to EEE2, wherein calculating the one or more sensed excitation levels comprises: Calculate the excitation level associated with the background component of the audio data; and Calculate the excitation level associated with the target component of the audio data.

[0133] EEE7. The method described in EEE6, which is subordinate to EEE5, wherein calculating the one or more sensed excitation levels further comprises: The excitation loss function is applied to the excitation level associated with the background component; and The excitation loss function is applied to the excitation level associated with the target component of the audio data.

[0134] EEE8. The method as described in any one of the preceding EEE methods further includes: Receive loudness adjustment data; and The gain is updated based on the loudness adjustment data.

[0135] EEE9. The method as described in any of the preceding EEEs, wherein the gain data is calculated and applied in two or more frequency bands.

[0136] EEE10. The method as described in any of the preceding EEEs, wherein the gain data is calculated and applied in a time-varying manner.

[0137] EEE11. The method as described in any one of the preceding EEEs, wherein calculating the gain data includes optimizing the perceived loudness constraint.

[0138] EEE12. The method as described in any of the preceding EEEs, wherein the context data includes ambient sound data; and wherein the ambient sound data is obtained from one or more microphones on a device close to the listener.

[0139] EEE13. The method as described in any of the preceding EEEs, wherein the hearing loss data is obtained based on the listener's age or sex at birth.

[0140] EEE14. The method as described in any of the preceding EEEs, wherein input loudness adjustment data is received at the playback device via user input from the listener.

[0141] EEE15. The method as described in any one of the preceding EEEs, wherein the playback device is a wearable device; and Perform one or more of the following on the wearable device: calculate audio level data, calculate gain data, and process the audio data using the gain data.

[0142] EEE16. The method as described in any of the preceding EEEs, wherein the contextual data includes ambient sound data; and wherein the ambient audio data is captured from one or more microphones of an ear-worn device communicating with a mobile device.

[0143] EEE17. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform a method as described in any one of EEE1 to EEE16.

[0144] EEE18. A computing device comprising: At least one processor; and The memory stores instructions that, when executed by the at least one processor, cause the computing device to perform a method as described in any one of EEE1 to EEE16.

[0145] EEE19. A method for processing audio, the method comprising: Receive audio data; Obtain hearing loss data associated with the listener; Receive context data including playback level data; Gain data is calculated based on the audio data, the hearing loss data, and the contextual data; and At least a portion of the audio data is processed using the gain data to produce compensated audio data.

[0146] EEE20. The method as described in EEE19, wherein the context data includes signal description data, and the method further includes: Determine whether the value associated with the signal description data is a first value or a second value, and Based on the determination that the value associated with the signal description data is a first value, the gain data is a first set of gain data; and The gain is a second set of gain data that is different from the first set of gain data, based on the determination that the value associated with the signal description data is a second value.

[0147] EEE21. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform a method as described in any one of EEE19 or EEE20.

[0148] EEE22. A computing device comprising: At least one processor; and The memory stores instructions that, when executed by the at least one processor, cause the computing device to perform a method as described in any one of EEE19 or EEE20.

Claims

1. A method for processing audio for a listener on a playback device, the method comprising: Receive audio data for playback on the playback device, the audio data including target audio components; Receive masked audio data; Obtain hearing loss data associated with the listener; Obtain playback level data; The target excitation mode is calculated based on the target audio components and the playback level data; The masking excitation pattern is calculated based on the masked audio data; The signal gain is calculated based on the target excitation mode, the masking excitation mode, and the hearing loss data. as well as The signal gain is applied to the target audio component to produce a compensated target audio component.

2. The method according to claim 1, wherein, The masked audio data includes background audio components contained in the audio data, and the masking excitation mode includes a background excitation mode based on the background audio components and the playback level data.

3. The method of claim 2, further comprising receiving descriptive data indicating whether a particular audio stream of the audio data is a target audio component or a background audio component.

4. The method according to claim 2, further comprising separating the audio data into the target audio component and the background audio component.

5. The method according to claim 4, wherein, The step of separating the audio data into the target audio component and the background audio component involves determining the target audio component of the audio data based on the direction or location of one or more audio objects or audio sources represented in the audio data.

6. The method according to any one of the preceding claims, wherein, The masking audio data includes ambient sound captured by a microphone near the listener, and the masking excitation pattern includes an ambient excitation pattern based on the ambient sound.

7. The method according to claim 6, wherein, The playback device includes a set of wearable devices, and the ambient sound is captured from the microphone of the wearable devices.

8. The method according to claim 6 or 7, further comprising: Receive playback device data, the playback device data describing the impact of the playback device on ambient sound, and wherein the ambient excitation mode is also based on the playback device data.

9. The method according to claim 8, wherein, The playback device includes a set of wearable devices having at least one earpiece, each earpiece including a microphone, a speaker, and an active filter, wherein the playback device data includes the transfer function of the active filter.

10. The method according to any one of the preceding claims, wherein, The signal gain is calculated and applied across two or more frequency bands.

11. The method according to any one of the preceding claims, wherein, The signal gain is calculated and applied in a time-varying manner.

12. The method according to any one of the preceding claims, wherein, Calculating the signal gain includes: Calculate the predicted loudness of the target excitation pattern under conditions of no masking and no hearing loss. Calculate a specific loudness of the target excitation pattern masked by the masked excitation pattern based on the hearing loss data. The signal gain is determined as the gain of the target audio component, such that the predicted specific loudness is approximately equal to the partial specific loudness.

13. The method according to claim 12, wherein, The signal gain is calculated as follows: the predicted loudness is calculated as: in, E t The target incentive mode, E m The background stimulus mode, E e The environmental stimulus mode is described above. A 0 represents the model parameter indicating 0 dB hearing loss, A ΔT α represents the parameters of the hearing loss-related model, and α is a constant between 0.1 and 0.

4.

14. The method according to any one of the preceding claims, further comprising: Loudness adjustment factor; as well as The signal gain is also calculated based on the loudness adjustment factor.

15. The method according to claim 14, wherein, The loudness adjustment factor is received at the playback device via user input from the listener.

16. The method according to claim 14 or 15, wherein, The signal gain is calculated as follows: the predicted loudness is calculated as: Wherein, λ is the loudness adjustment factor. E t The target incentive mode, E m The background stimulus mode, E e The environmental stimulus mode is described above. A 0 represents the model parameter indicating 0 dB hearing loss, A ΔT α represents the parameters of the hearing loss-related model, and α is a constant between 0.1 and 0.

4.

17. The method according to any one of the preceding claims, wherein, Hearing loss data were obtained based on the listener's age or sex at birth.

18. The method according to any one of the preceding claims, wherein, The playback device includes a set of wearable hearing aids; and Specifically, one or more of the following are performed on the hearing-wearable device: calculating signal gain and applying the signal gain.

19. A method for audio rendering, the method comprising: Receive an audio bitstream that includes a target audio component and a background audio component; Receive audio scene information, which includes the listener's current head posture and / or gaze; Process the audio scene information to obtain the audio scene state; The audio data is rendered based on the scene state to obtain the rendered target audio component and the rendered background audio component; Obtain hearing loss data associated with the listener; Obtain playback level data; The target excitation mode is calculated based on the target audio components and the playback level data; The masking excitation mode is calculated based on the background audio components and the playback level data. The signal gain is calculated based on the target excitation mode, the masking excitation mode, and the hearing loss data. as well as The signal gain is applied to the rendered target audio component to produce a compensated rendered target audio component.

20. The method of claim 19, further comprising receiving descriptive data indicating whether a particular audio stream of the audio data is a target audio component or a background audio component.

21. The method according to claim 19, wherein, At least some of the target audio components and / or background audio components are identified based on the current head posture and / or gaze.

22. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform the method as claimed in any one of claims 1 to 21.

23. A computing device, comprising: At least one processor; as well as A memory that stores instructions that, when executed by the at least one processor, cause the computing device to perform the method as described in any one of claims 1 to 21.

24. An audio renderer, comprising: A decoder, which receives an audio bitstream and decodes the audio bitstream to provide input audio including a target audio component and a background audio component; A scene controller, configured to receive audio scene information and provide an audio scene status based on the audio scene information; A rendering pipeline, configured as follows: The audio data is rendered based on the scene state to obtain the rendered target audio component and the rendered background audio component; Obtain hearing loss data associated with the listener; Obtain playback level data; The target excitation mode is calculated based on the target audio components and the playback level data; The masking excitation mode is calculated based on the background audio components and the playback level data. The signal gain is calculated based on the target excitation mode, the masking excitation mode, and the hearing loss data. as well as The signal gain is applied to the rendered target audio component to produce a compensated rendered target audio component.

25. The audio renderer according to claim 24, wherein, The rendering pipeline is further configured to receive descriptive data indicating whether a particular audio stream of the audio data is a target audio component or a background audio component.

26. The audio renderer according to claim 24 or 25, wherein, The audio renderer is compatible with the MPEG-I standard.

Images