System and method for providing personalized audio playback on multiple consumer devices

Abstract

A system and method for processing audio signals for server-adjusted sound personalization on multiple consumer devices is provided. A user hearing test is performed on one of multiple audio output devices. Next, hearing data from the user hearing test is output to a server and stored on a database of the server along with a unique user identifier. Next, a set of DSP parameters for a sound personalization algorithm is calculated based on the user's hearing data. Then, when the user logs into an application on the audio output device with his unique identifier, the set of DSP parameters is output to one of the multiple audio output devices.

Description

Technical Field

[0001] The present invention relates generally to the fields of digital signal processing (DSP), audio engineering, and audiology - and more particularly to systems and methods for providing server-modulated sound personalization on multiple consumer devices based on user hearing test results. Background Art

[0002] Traditional DSP sound personalization methods typically rely on the management of audiograms to parameterize frequency gain compensation functions. Typically, a pure tone threshold (PTT) audiometry test is employed to identify frequencies at which the user exhibits elevated hearing thresholds and modulate the frequency output accordingly. These gain parameters are stored locally on the user's device for subsequent audio processing.

[0003] However, this method of enhancing the sound experience for users is imprecise and inefficient. Because the hearing test results are stored locally on a single device, the resulting parameter calculations are not accessible to a central server and other devices. To this extent, a separate hearing test must be performed on each device - which can result in local errors in the results and inconsistent parameter values ​​stored on different audio output devices. The ability to perform hearing tests on multiple devices associated with a core account: 1) encourages users to test on whichever paired device is most convenient at the time; 2) improves accuracy by merging multiple test results; and 3) enables tracking of the user's hearing status over time. In addition, in the event of an abnormal hearing test result, the user can be informed whether he or she used an improperly calibrated device and / or whether the hearing test was performed improperly.

[0004] To the extent that simply applying a gain function to an audio signal is insufficient to restore audibility, the use of frequency compensation is even less effective. The gain may enable a user to recapture previously unheard frequencies, but the user may then experience loudness discomfort. Listeners with sensorineural hearing loss typically have similar or even reduced thresholds for discomfort when compared to listeners with normal hearing, despite their improved hearing thresholds. To this extent, their dynamic aperture will be narrowed, and simply increasing gain will be detrimental to their hearing health in the long run.

[0005] Although hearing loss often begins in the higher frequencies, listeners do not usually complain about the presence of high-frequency sounds when they become aware of their hearing loss. Instead, they report difficulty listening in noisy environments and hearing details in complex sound mixtures, such as an audio stream of a radio interview on a busy street. Essentially, for a hearing-impaired (HI) person, cutoff frequency sounds are more likely to mask information with energy from other frequencies - music that was once clear and detailed becomes muddled. This is because music itself is highly self-masking, i.e., the energy of many sound sources overlaps in frequency space, which can reduce direct detection capabilities, or hinder the user's ability to extract information from certain sound sources.

[0006] As hearing deteriorates, the ear's signal conditioning ability begins to break down, so that HI listeners need to expend more effort to perceive sounds of interest in complex acoustic scenarios (or miss the information entirely). An increase in audiogram thresholds is not simply a decrease in hearing sensitivity, but is the result of a malfunction in some deeper process in the auditory system that has an impact beyond detecting faint sounds. To this extent, simple increases in frequency gain provide an inadequate solution.

[0007] Therefore, one aspect of the present disclosure is to provide a system and method for providing personalized audio playback on multiple consumer devices through a server-authorized sound personalization account. By providing a more accurate and portable parameter set, users can enjoy sound personalization across a large number of devices through a simple hearing test, thereby enjoying a healthier listening experience. Summary of the invention

[0008] According to one aspect of the present disclosure, a system and method for providing personalized audio playback on multiple consumer devices is provided. In some embodiments, the system and method include steps including: performing a user hearing test on one of multiple audio output devices, outputting hearing data from the user's hearing test to a server; storing the hearing data with a user unique identifier on a database of the server; calculating a set of parameters for a sound personalization algorithm based on the user hearing data, and storing the parameters on the database together with the unique user identifier; outputting the set of parameters to the sound personalization algorithm on one of multiple audio output devices, wherein the parameters are output when the user enters his unique identifier on one of the audio output devices; and processing an audio signal on one of the audio output devices using the sound personalization algorithm.

[0009] In another embodiment, the system and method include steps including: performing a user hearing test on one of a plurality of audio output devices; calculating a set of parameters for a sound personalization algorithm on the audio output device based on the user's hearing data; outputting the hearing data and the calculated parameters to a server; storing the hearing data and the calculated parameters with a unique identifier on a database of the server; outputting the set of parameters to the sound personalization algorithm on one of a plurality of audio output devices, wherein the parameters are output when the user inputs his or her unique identifier into an application running on one of the audio output devices; and processing an audio signal on one of the audio output devices using the sound personalization algorithm.

[0010] In another embodiment, the system and method include steps including: performing a user hearing test on one of a plurality of audio output devices; outputting hearing data of the user's hearing test to a server; storing the hearing data and a unique identifier on a database of the server; outputting the hearing data to one of a plurality of audio output devices, wherein the hearing data is output when the user inputs his / her unique identifier into an application running on one of the audio output devices; calculating a set of parameters for a sound personalization algorithm on the audio output device based on the user hearing data; and outputting the parameters to the sound parameterization algorithm on the audio output device.

[0011] In some embodiments, the hearing test is one or more of a threshold test, a suprathreshold test, a psychophysical tuning curve test, a masked threshold test, a temporal fine structure test, a speech-in-noise test, and a temporal masking curve test.

[0012] In some embodiments, the parameters are recalculated when the user performs an additional hearing test on any of the plurality of audio output devices. The additional hearing test may reflect information from another critical band. The additional hearing test may be a different type of hearing test than the previously performed hearing test, or it may be an updated version of the previously performed hearing test. Additionally, the additional hearing test may replace hearing data from the previously performed hearing test with an abnormal result.

[0013] In some embodiments, the set of parameters is calculated on demand on the server when the user enters their unique identifier on one of the audio output devices.

[0014] In some embodiments, the hearing test measures the masking threshold curve over a frequency range from 250 Hz to 12 kHz.

[0015] In some embodiments, the sound personalization algorithm operates on subband signals of the audio signal, i.e., the algorithm works selectively according to frequency. In another embodiment, the parameters of the sound personalization algorithm include at least one of a gain value provided in each subband and a limiter value provided in each subband. In an optional embodiment, the sound personalization algorithm may be a multi-band dynamic processing algorithm. The parameters of the multi-band dynamics processor may optionally include at least one of a threshold value of a dynamic range compressor provided in each subband, a ratio value of a dynamic range compressor provided in each subband, and a gain value provided in each subband. In an alternative embodiment, the parameters of the multi-band dynamics processor may optionally be determined according to a hearing aid gain table, which specifies the amount of gain for a given input sound level at each frequency. Typically, these tables include columns for the frequency of the sound input and rows for different intensities of the sound input (which may also be mapped to threshold, ratio and gain settings in general).

[0016] In some embodiments, the parameters are calculated indirectly. For example, the parameters may be calculated using a best fit of the user's hearing data to a previously entered entry in the server database, wherein the parameters associated with the best fit, previously entered hearing data are copied and entered into the server database entry for the user. For example, the best fit may be determined by measuring the average Euclidean distance between the user's hearing data and the hearing data in the database. Alternatively, the root mean square distance or similar best fit measure may be used.

[0017] In some embodiments, parameters may be calculated using a closest fit of the user hearing data to at least two hearing data entries in the server database, wherein parameters associated with the closest fit, previously input hearing data are interpolated and input into the server database entry for the user. For example, parameters may be interpolated linearly between two parameter values. Alternatively, parameters may be interpolated nonlinearly, such as by a square function. Alternatively, the parameters may be calculated based on a fitted mathematical function, such as a polynomial function, derived from plotting existing hearing and parameter set data entries in the server database.

[0018] In some embodiments, the parameters may be calculated by converting the user's hearing test results into a 'hearing age' value. Based on the predictable decline in hearing function, the user's hearing test results may be matched to the closest representative hearing age. From there, the hearing age parameter set is then copied into the user's hearing curve.

[0019] In some embodiments, the parameters may be calculated directly. For example, the parameters may be calculated by fitting a user masking profile curve to a target masking profile curve. Alternatively, the parameters may be calculated by optimizing perceptually relevant information (PRI). Alternatively, the parameters may be calculated using canonical techniques known in the art.

[0020] In some embodiments, the consumer electronic device is one of a mobile phone, a tablet computer, a television, a desktop computer, a laptop computer, a hearable, a smart speaker, a headset, and a speaker system.

[0021] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this technology belongs.

[0022] As used herein, the term "sound personalization algorithm" is defined as any digital signal processing (DSP) algorithm that processes an audio signal to enhance the signal clarity to a listener. The DSP algorithm may be, for example, an equalizer, an audio processing function operating at the sub-band level of an audio signal, a multi-band compression system, or a non-linear audio processing algorithm.

[0023] As used herein, the term "audio output device" is defined as any device that outputs audio, including but not limited to: mobile phones, computers, televisions, hearing aids, headphones, smart speakers, hearables, and / or speaker systems.

[0024] As used herein, the term "headphones" is any earpiece with a transducer that outputs sound waves into the ear. The earphones can be wireless hearables, wired or wireless headphones, hearables, or any pair of earbuds.

[0025] As used herein, the term "hearing test" is any test that assesses the hearing health of a user, more specifically a hearing test performed using any transducer that outputs sound waves. The test may be a threshold test or a suprathreshold test, including but not limited to a psychophysical tuning curve (PTC) test, a masking threshold (MT) test, a temporal fine structure test (TFS), a temporal masking curve test, and a speech in noise test.

[0026] As used herein, the term "server" generally refers to a computer program or device that provides functionality to other programs or devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to describe the manner in which the above-recited and other advantages and features of the present disclosure can be obtained, a more particular description of the principles briefly described above will be given by reference to specific embodiments of the present disclosure, which are illustrated in the accompanying drawings. Understanding that these drawings depict only exemplary embodiments of the present disclosure and are therefore not to be considered limiting of its scope, the principles herein will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0028] Figure 1 A flow chart showing a server receiving hearing test data input from a plurality of devices and outputting sound personalized DSP parameters to a plurality of audio output devices;

[0029] Figure 2A and Figure 2B An exemplary graph illustrating the deterioration of human audiogram and masking threshold with age is shown;

[0030] Figure 3A and Figure 3B An example of a PTC and MT test paradigm is shown;

[0031] Figure 4 A method for server-side management of personalized audio playback on multiple consumer devices in accordance with one or more aspects of the present disclosure is shown;

[0032] Figure 5 A method for server-side management of personalized audio playback on multiple consumer devices in accordance with one or more aspects of the present disclosure is shown;

[0033] Figure 6 A method for server-side management of personalized audio playback on multiple consumer devices in accordance with one or more aspects of the present disclosure is shown;

[0034] Figure 7 A method for server-side management of personalized audio playback is shown, wherein a user's parameter settings are updated based on multiple hearing tests conducted over a period of time;

[0035] Figure 8 A method for calculating sound personalization parameters using threshold and suprathreshold tests is shown;

[0036] Fig. 9 conceptually illustrates masking threshold curve widths for three different users, which may be used in best fit and / or closest fit calculations;

[0037] Fig.10 conceptually illustrating an audiogram graph of three different user x, y, and z data points that may be used for best fit and / or closest fit calculations;

[0038] Fig.11 A method for parameter calculation using a best fit method is shown;

[0039] Fig.12 A method for parameter calculation using interpolation of the closest fit to the hearing data is shown;

[0040] Fig.13 A method of obtaining ratio and threshold parameters from a user masking profile curve is shown;

[0041] Fig.14 shows a graph for obtaining ratio and threshold parameters from a user PTC curve;

[0042] Fig.15 A method for obtaining DSP parameters from user hearing data by optimizing perceptually relevant information is shown;

[0043] Fig.16 Exemplary system embodiments are shown in which one or more aspects of the present disclosure may be employed. DETAILED DESCRIPTION

[0044] Various embodiments of the present disclosure are discussed in detail below. Although specific implementations are discussed, it should be understood that this is done for illustrative purposes only. Those skilled in the relevant art will recognize that other components and configurations may be used without departing from the spirit and scope of the present disclosure. Therefore, the following description and drawings are illustrative and should not be construed as limiting the scope of the embodiments described herein. Many specific details are described to provide a thorough understanding of the present disclosure. However, in some cases, well-known or conventional details are not described in order to avoid making the description unclear. References to one / an embodiment in the present disclosure may be references to the same embodiment or any embodiment; and such references represent at least one of the embodiments.

[0045] Reference to "one embodiment" or "an embodiment" indicates that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appearing in various places in the specification does not necessarily all refer to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. In addition, various features are described that may be exhibited by some embodiments but not by others.

[0046] The terms used in this specification generally have their ordinary meanings in the art, in the context of the present disclosure, and in the specific context in which each term is used. Alternative language and synonyms may be used for any one or more of the terms discussed herein, and there is no special meaning as to whether the terms are elaborated or discussed herein. In some cases, synonyms for certain terms are provided. The description of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification (including examples of any terms discussed herein) is merely illustrative, and is not intended to further limit the scope and meaning of the present disclosure or any exemplary term. Likewise, the present disclosure is not limited to the various embodiments provided in this specification.

[0047] Without intending to limit the scope of the present disclosure, examples of instruments, devices, methods and related results thereof according to embodiments of the present disclosure are given below. Note that for the convenience of the reader, titles or subtitles may be used in the examples, which should in no way limit the scope of the present disclosure. Unless otherwise defined, the technical and scientific terms used herein have the meanings commonly understood by those of ordinary skill in the art to which the present disclosure belongs. In the event of a conflict, the present document including the definitions shall prevail.

[0048] Additional features and advantages of the present disclosure will be set forth in the following description and will be partially apparent from the description, or may be learned by practicing the principles disclosed herein. The features and advantages of the present disclosure may be realized and obtained by the instruments and combinations specifically pointed out in the appended claims. These and other features of the present disclosure will become more apparent from the following description and the appended claims, or may be learned by practicing the principles set forth herein.

[0049] Various exemplary embodiments of the present disclosure are discussed in detail below. Although specific implementations are discussed, it should be understood that this is done for illustrative purposes only. Those skilled in the relevant art will recognize that other components and configurations may be used without departing from the spirit and scope of the present disclosure.

[0050] The systems and methods according to various aspects of the present disclosure solve the problem of accurately and effectively personalizing the audio output of multiple audio output devices, and in particular solve the problem of personalizing in a consistent manner. By implementing server-adjusted sound personalization based on user hearing test results, various aspects of the present disclosure provide seamless sound personalization between various audio output devices. To this extent, users' greater adoption of sound personalization and / or enhancement will not only provide users with a richer and clearer listening experience, but also make user behavior healthier when listening to audio.

[0051] like Figure 1As shown in , a user may, for example, perform a hearing test on a mobile phone 101, a laptop computer 102, or in front of a television 109, and the results of the hearing test may then be output to a server 103. The resulting DSP parameters may then be calculated and stored on the server 103. This DSP parameter calculation may be performed on the initial device (e.g., 101, 102, 103) where the hearing test is performed, on the server 103, or on some combination thereof. The resulting DSP parameters may then be output to a plurality of end-user devices, including but not limited to a media player 104, a mobile phone 105, a smart speaker 106, a laptop computer 107, and a television 108, etc. The calculated DSP parameters are then used to provide seamless and consistent sound personalization across each of the various end-user devices 104-108. In some embodiments, the user's hearing test data is output to one or more of the end-user devices 104-108, so that the DSP parameter calculation may be performed on the end-user device. In some embodiments, one or more of the end-user devices 104 - 108 may be employed to perform the user hearing test, ie, the initial device 101 - 103 and the end-user device 104 - 108 may be provided as a single device.

[0052] Figure 2A and Figure 2B Emphasizing the importance of sound personalization, it is shown that a listener's hearing ability deteriorates over time. Starting at the age of 20, humans begin to lose their ability to hear higher frequencies - Figure 2A (albeit above the spectrum of human speech). This loss progressively worsens with age, as a clear decline within the speech spectrum becomes apparent around age 50 or 60. However, these pure tone audiometry findings mask a more complex problem, as the ability to understand speech can decline much earlier. Although hearing loss often begins at the higher frequencies, listeners typically do not complain about the presence of high frequency sounds when they become aware of their hearing loss. Instead, they report difficulty listening in noisy environments and hearing details in complex sound mixtures (such as in a phone call). Essentially, cutoff frequency sounds more easily mask frequencies of interest to the hearing-impaired person - conversations that were once clear and detailed become muddled. As hearing deteriorates, the ear's signal conditioning ability begins to break down, so the hearing-impaired listener needs to expend more effort to perceive sounds of interest in complex acoustic scenes (or miss the information entirely). An increase in audiogram thresholds is not simply a decrease in hearing sensitivity, but the result of a malfunction in some deeper process in the auditory system that has an impact beyond detecting faint sounds.

[0053] To this extent, Figure 2BThe key, discernible age trends in suprathreshold hearing tests are shown. The psychophysical tuning curve (PTC) test 202 is a suprathreshold test that measures an individual's ability to discern a probe tone (or pulse signal tone) against a swept masking noise of variable frequency and amplitude. For example, the psychophysical tuning curve test can be measured for a signal tone between 500 Hz and 4 kHz frequency and at a probe tone level between 20 dB SL and 40 dB SL in the presence of a masking signal for the signal tone, the masking signal sweeping from 50% of the signal tone frequency to 150% of the signal tone frequency. In addition, although the level of the probe tone can be between 20 dB SL and 40 dB SL (although various other ranges such as 5 dB SL-40 dB SL are also possible), it should be noted that the level of the swept masking signal may exceed, and even greatly exceed, the sound level range associated with the probe tone. By collecting a large data set, key age trends 201 can be determined, allowing accurate parameterization of personalized DSP algorithms. For example, in a multi-band compression system, the threshold and ratio values ​​of each sub-band signal dynamic range compressor (DRC) can be modified to reduce the problem area of ​​frequency masking, and the compressed sub-band signal gain can be further applied to the relevant area. Figure 2A and Figure 2B In the context of , the masking threshold curve 202 represents a similar example for measuring the masking threshold. The narrow noise band (in this case 203, approximately 4kHz) is fixed while the probe tone is swept from 50% of the center frequency of the noise band to 150% of the center frequency of the noise band. Similarly, large MT data sets can be collected to determine key age trends.

[0054] Figure 3A and Figure 3B An exemplary method in which a PTC test 301 or MT test 305 can be performed is shown. A psychophysical tuning curve (PTC) consisting of a frequency selectivity profile 304 extracted via a behavioral test provides useful data to determine an individual's masking profile. In one embodiment of the test, a masking noise band 302 is gradually swept across a frequency from below a probe tone frequency 303 to above a probe tone frequency 303. Then, the user responds when they can hear the probe tone, and stops responding when they no longer hear the probe tone. This gives a sawtooth track, which can then be interpolated to estimate the basic characteristics of an auditory filter by a masking profile graph. Other methods known in the prior art can be used to obtain a user masking profile curve. For example, a counterexample can be used, in which a probe tone 306 is swept across frequencies while a masking noise band 307 is fixed at a center frequency (referred to as a "masking threshold test" or "MT test").

[0055] Figure 4An exemplary embodiment of the present disclosure is shown, wherein personalized audio playback is performed on multiple audio output devices. First, a hearing test is performed 407 on one of the multiple audio output devices. The hearing test can be provided by any of a variety of hearing test options, including but not limited to: masked threshold test (MT test) 401, pure tone threshold test (PTT test) 402, psychophysical tuning curve test (PTC test) 403, time fine structure test (TFS test) 404, speech in noise test 405 or other suprathreshold test 406.

[0056] Next, the hearing test results are output 408 to the server along with one or more of a timestamp and a unique user identifier. DSP parameters for the sound personalization algorithm are then calculated and stored 409 in a server database. The calculated DSP parameters for a given sound personalization algorithm may include, but are not limited to: ratios, thresholds, and gain values ​​in a multi-band dynamics processor, gain and limiter values ​​for an equalization DSP, and / or parameter values ​​common to other sound personalization DSPs (see, e.g., commonly owned U.S. Pat. No. 10,199,047 and U.S. Patent Application No. 16 / 244,727, the contents of which are incorporated herein by reference in their entirety).

[0057] As explained below, one or more of the DSP parameter calculations may be performed directly or indirectly. When a user enters 410 their unique identifier on one of the plurality of audio output devices, the user's DSP parameters are retrieved 411 from a server database and output 412 to the sound personalization DSP of the audio output device. The user's unique identifier may be entered into a standalone application on the user's device - or alternatively or additionally, it may be entered into an existing application along with a plug-in login functionality that adjusts server connectivity. After the user's unique identifier has been received and the corresponding DSP parameters retrieved from the server database, the audio signal is then processed 413 locally on the given audio output device using the parameterized sound personalization DSP.

[0058] Figure 5 Shows Figure 4 An alternative embodiment of the method shown in . Figure 5 , a hearing test (e.g., selected from a set of hearing tests 501-506, which in some embodiments may be the same as the set of hearing tests 401-406) is first performed 507 on one of a plurality of audio output devices. After performing the hearing test, the audio output device then calculates 508 the DSP parameters itself and outputs 509 these locally calculated DSP parameter values ​​and the user's hearing data to a server database. This output is then stored 509 in the server database along with the user's unique identifier.

[0059] and Figure 4 Compared to an embodiment of the present invention (in which the server receives the hearing test data of the user and then performs server-side calculation of the DSP parameters), Figure 5 Embodiments of the present invention perform local calculation of the DSP parameters (i.e., the local calculation is performed on the same audio output device where the hearing test is performed / measured). However, because the audio output device transmits both the locally calculated DSP parameters and the underlying hearing test data to the server, it is still possible to perform server-side calculation of the DSP parameters. For example, such server-side calculation can be used to verify or otherwise enhance the local calculation, or as a backup measure.

[0060] Then, similar to Figure 4 In an embodiment of the present invention, when a user enters 510 his unique identifier in an application on one of the plurality of audio output devices, the user's DSP parameters are retrieved 511 from a server database and output 512 to the sound personalization DSP of the audio output device. The audio signal is then processed 513 accordingly.

[0061] Figure 6 Shows Figure 4 and Figure 5 Another alternative embodiment of the method shown in . Here, a hearing test is performed 607 on one of a plurality of audio output devices, and the hearing test data is then stored 608 on a server database along with a unique user identifier. Figure 6 In an embodiment of the present invention, the DSP parameters for the hearing test data are not necessarily calculated in response to receiving the hearing test data at the server or server database. It is worth noting that instead, the DSP parameters can be calculated 611 on demand or just in time on the end-user device, on the server, or some combination of the two, where the calculation 611 is based on the hearing data retrieved 610 from the server database when the user enters 609 his unique identifier on the end-user device. The calculated DSP parameters can optionally be output to the server to be stored with the user's hearing data.

[0062] Importantly, in Figure 4-Figure 6 The previously discussed methods shown in generally feature a central server that coordinates the exchange of data required for sound personalization across multiple audio output devices, regardless of when and where the calculations of DSP parameter values ​​are performed.

[0063] Figure 7 Another exemplary embodiment is shown, demonstrating how the user-calculated DSP parameters may vary over time (e.g., in this case, as the user performs multiple hearing tests). Figure 7As depicted in FIG. 1 , at t1, the user performs MT test 701 using a 4 kHz noise probe tone, and then, for example, according to Figure 4-Figure 6 Any of the methods shown in calculate and store DSP parameters based on hearing data from an MT test. Next, at t2, the user takes an MT test 702 using a 1 kHz noise probe tone, representing hearing data within another region of the individual's auditory filters. Combined with the data collected at t1, the combination of these two data sets can provide a more comprehensive overview of the user's hearing health, thereby enabling more accurate calculation of DSP parameters. To this extent, the DSP parameters are updated based on the t1 and t2 data. The user can then take further tests, such as a pure tone threshold test 703 or an MT test 704 using a 2 kHz noise probe tone, to further refine their hearing profile. Additionally, the user can retake the test 705 at a later time point t5, which may indicate that an earlier test result was abnormal and / or that the user's hearing has declined since the previous test - both of which result in the parameters being updated based on the new results. Overall, Figure 7 The method shown in provides an evolving and accurate hearing profile that informs updated calculations for the sound personalization algorithm.

[0064] Figure 8 Further illustrated are exemplary embodiments for personalizing audio playback in accordance with aspects of the present disclosure. Hearing test data 801 is obtained from a user and used to calculate sound personalized DSP parameters 803, in this case for a multi-band dynamics processor using at least parameter value ratios (r), thresholds (t), and gains (g) for each subband 1 to x. Here, the hearing test data 801 is provided as MT and PTT data, although other types of hearing test data may be utilized without departing from the scope of the present disclosure. Within the server, the hearing data may be stored as exemplary entries [(u_id), t, MT, PTT] 802, where u_id is a unique user ID, t is a timestamp, MT is hearing data associated with a MT test, and PTT is hearing data associated with a PTT test. Additionally, when calculating a set of DSP parameters for a given sound personalization algorithm, they may be provided as [DSP q-param ] is added to entry 804. When the user subsequently enters his u_id to log into a given application on his audio output device 805, the [DSP q-param ]806 is then output to the audio output device 807. Figure 8 In the example, parameter set values ​​803 contain at least ratios and thresholds and gain values ​​for a dynamic range compressor for each subband signal from 1 to x in a multi-band dynamics processor sound personalization algorithm.

[0065] In some embodiments, the DSP parameter set may be calculated directly from the user's hearing data or indirectly based on pre-existing entries or anchor points in a server database. Anchor points include typical hearing curves constructed at least in part based on demographic information such as age and gender, where the DSP parameter set is calculated and stored on the server for use as a reference marker. Indirect calculation of the DSP parameter set bypasses direct parameter set calculation by finding the closest matching hearing curve and importing (or interpolating) these values ​​for the user.

[0066] Fig. 9 Three conceptual user masking threshold (MT) curves for users x, y, and z are shown. The MT curves are centered at frequencies ad, each having a curve width d, which can be used as a measure of the similarity between user hearing data. For example, a root mean square difference calculation can be used to determine whether user y's hearing data is more similar to user x or user z's hearing data, for example, by the following calculation:

[0067]

[0068] Fig.10 Three conceptual audiograms are shown for users x, y, and z, each with pure tone thresholds 1 - 5. Similar to the above, the RMS difference measure can also be used to determine, for example, whether the hearing data of user y is more similar to that of user x than that of user z, for example, by the following calculation:

[0069]

[0070] As will be appreciated by one of ordinary skill in the art, other methods may be used to quantify the similarity between user hearing curve graphs, which may include, but are not limited to, methods such as Euclidean distance measures, e.g., ((y1-x1)+(y2-x2)...>(y1-x1)+(y2-x2))... or other statistical methods known in the art. Then, for indirect DSP parameter set calculations, the closest matching hearing curve between the user and other pre-existing database entries or anchor points may then be used.

[0071] Fig.11 An exemplary embodiment for calculating a set of sound personalization parameters for a given algorithm based on pre-existing entries and / or anchor points is shown. Here, the server database entries 1102 are investigated to find the best fit to the user hearing data input 1101, which is represented as for (u_id) 200 MT 200 and PTT 200 This can be done by Fig. 9 and Fig.10The statistical techniques shown in Fig.11 In the example, (u_id) 200 The hearing data is best matched to the MT3 and PTT3 data 603. To this extent, then the parameter set associated with (u_id)3 [DSP q-param 3] for (u_id) 200 Parameter set entries, shown here as [(u_id) 200 ,t 200 ,MT 200 ,PTT 200 ,DSP q-param 3].

[0072] Fig.12 An exemplary embodiment for calculating a set of sound personalization parameters for a given algorithm based on pre-existing entries or anchor points according to aspects of the present disclosure is shown. Here, the server database entry 1202 is used to interpolate 1204 between the entries for (u_id) 300 User hearing data input 1201MT 300 and PT 300 The two closest fits are between 1200. In this example, (u_id) 300 The hearing data were most closely fitted between: and To this extent, the (u_id)3 and (u_id)5 parameter sets are interpolated to produce the value for (u_id) 300 The new parameter set of the parameter set entry is represented here as [(u_id) 200, t 200 ,MT 200, PTT 200 ,DSP q-param3 / 5 ]705. In another embodiment, interpolation may be performed across multiple data entries to calculate sound personalization parameters, such as /

[0073] The DSP parameter sets can be linearly interpolated, for example, in Fig.12 In the example of FIG. 1 , assuming that the hearing data of user 200 is between the hearing data of user 3 and the hearing data of user 5, the DRC ratio value 0.7 for user 5 (u_id) 5 and the DRC ratio value 0.8 for user 3 (u_id) 3 will be interpolated as the DRC ratio value of user 200 (u_id) 200 The DRC ratio value is 0.75. In some embodiments, the DSP parameter set can also be nonlinearly interpolated, for example, using a square function, for example, Fig.12 In the example of , the DRC ratio value of 0.6 for user 5 and the DRC ratio value of 0.8 for user 3 will be nonlinearly interpolated to a DRC ratio value of 0.75 for user 200.

[0074] Fig.13 , Fig.14 and Fig.15 Various exemplary methods for directly calculating a set of parameters based on user hearing data according to one or more aspects of the present disclosure are shown. In one embodiment, this can be accomplished using a hearing aid gain table specification formula. In another embodiment, the ratios and thresholds and gains for a compressor in a given multi-band dynamic processor signal subband can be calculated by comparing the user threshold and super-threshold information of the listener with the user threshold and super-threshold information of a normal hearing person (i.e., reference audiogram and PTC / MT curves). For example, masking contour curve data such as PTC or MT can be used to calculate ratios and threshold parameters for a given frequency subband, while audiogram data can be used to calculate gains within a given frequency subband.

[0075] Fig.13 and Fig.14 One way to configure the ratio and threshold parameters of the frequency bands in a multi-band compression system is shown (e.g., see commonly owned applications EP18200368.1 and US16 / 201,839, the contents of which are incorporated herein by reference). In short, a user's masking profile curve 1301 is received, a target masking curve 1302 is determined, and then the target masking curve is compared with the user masking profile curve 1301 to determine and output a user-calculated set of DSP parameters 1304.

[0076] Fig.14 A visualization combining a user masking contour curve 1406 for a listener (listener) and a target masking contour curve 1407 for a probe tone 1450 (the x-axis 1401 is frequency and the y-axis 1402 is sound level in dB SPL or HL), with an input / output graph of the compressor showing the input level 1403 of the sound signal versus the output level 1404 in decibels relative to full scale (dB FS). The bisector in the input / output graph represents the 1:1 (unprocessed) output for an input signal with a gain of 1.

[0077] The parameters of the multi-band compression system in the frequency band are the threshold 1411 and the gain 1412. These two parameters are determined based on the user masking profile curve 1406 and the target masking profile curve 1407 for the listener. The threshold 1411 and the ratio 1412 must meet the following conditions: the signal-to-noise ratio 1421 (SNR) of the user masking profile curve 1406 at a given frequency 1409 is greater than the SNR 1422 of the target masking profile curve 1407 at the same given frequency 1409. It should be noted that SNR is defined herein as the ratio of the level of the signal tone to the level of the masking noise. The wider the curve, the greater the SNR. The given frequency 1409 when calculating the SNR 1421 and 1422 can be arbitrarily selected, for example, so that it exceeds the minimum distance from the detection tone frequency 1408.

[0078] The target masking profile curve 1407 corresponds to the sound level 1430 (in dB) at a given frequency (see Fig.14 1431) into the input sound level 1441 of the compression system. The goal is that the sound level 1442 output by the compression system will match the user masking contour curve 1406, that is, the sound level 1442 is substantially equal to the sound level (in dB) of the user masking contour curve 1406 at a given frequency 1409. This condition allows the derivation of a threshold value 1411 (which must be lower than the input sound level 1441) and a ratio 1412. In other words, the input sound level 1441 and the output sound level 1442 determine the reference point of the compression curve. As described above, the threshold value 1411 must be selected to be lower than the input sound level 1441 - if not, no change will occur because the system is linear below the threshold of the compressor). Once the threshold value 1411 is selected, the ratio 1412 can be determined based on the threshold value and the reference point of the compression curve.

[0079] In the context of the present invention, a masking contour curve is obtained from a user hearing test. A target masking contour curve 1407 is interpolated from at least the user masking contour curve 1406 and a reference masking contour curve representing the curve for a normal hearing person. The target masking contour curve 1407 is preferred over the reference curve because fitting the audio signal to the reference curve is not necessarily optimal. Depending on the initial hearing of the listener, fitting the processing according to the reference curve may result in excessive processing, thereby destroying the signal quality. The goal is to process the signal so as to obtain a good balance between the target interest and good sound quality.

[0080] A given frequency 1409 is then selected. The given frequency may be arbitrarily selected, for example, a certain distance from the tone frequency 1408. The corresponding sound levels of the listener and target masking contour curves are determined at the given frequency 1409. The values ​​of these sound levels may be determined graphically on the y-axis 1402.

[0081] Fig.14The right panel in (see continuous graph) shows a hard knee DRC, with a threshold 1411 and a ratio 1412 as parameters to be determined. An input sound signal having a sound level 1430 / 1441 at a given frequency 1409 enters the compression system (see the curved arrow 1431 indicating the corresponding relationship between 1430 / 1441). The sound signal should be processed by the DRC in the following manner: the output sound level is the sound level of the user masking contour curve 1406 at a given frequency 1409. The threshold 1411 should not exceed the input sound level 1441, otherwise compression will not occur. There may be multiple threshold and ratio parameter sets. Depending on the fitting algorithm and / or target fitting data that have been proven to show the greatest benefit in terms of sound quality, a preferred set can be selected. For example, either of the threshold 1411 and the ratio 1412 can be selected to have a default value, and then the corresponding other of the parameters can be determined by applying the above conditions.

[0082] To calculate the gains within the subband signals, the results of the audiogram can be used. For example, an increased threshold can be compensated by a corresponding frequency gain.

[0083] In one embodiment of the present disclosure, Fig.15 As shown in , DSP parameters in a multi-band dynamics processor can be calculated by optimizing perceptually relevant information (e.g., perceptual entropy) by parameterizing using user threshold and suprathreshold hearing data (see commonly owned applications US16 / 206,376 and EP18208020.0). In short, in order to optimally parameterize the multi-band dynamics processor by perceptually relevant information, an audio sample 1501 or a body of audio samples is first processed 1502 by a parameterized multi-band dynamics processor, and the perceptual entropy of the file is calculated 1503 based on the user threshold and suprathreshold hearing data 1507. After the calculation, the multi-band dynamics processor is re-parameterized 1511 according to a given set of parameter heuristics derived from the optimization, and the audio samples are re-processed 1502 and the PRI calculated 1503 accordingly. In other words, the multi-band dynamics processor is configured to process the audio sample so that the audio sample has a higher PRI value for a particular listener when the threshold and suprathreshold information 1507 of the individual listener is taken into account. To this end, the parameterization of the multiband dynamics processor is adapted to increase the PRI of the processed audio samples relative to the unprocessed audio samples.The parameters of the multiband dynamics processor are determined by an optimization process using the PRI as its optimization criterion.

[0084] PRI can be calculated according to a variety of methods that have been discovered. James D. Johnston developed such a method at Bell Labs, also known as perceptual entropy, which generally includes: transforming a sampling window of the audio signal into the frequency domain, obtaining a masking threshold using psychoacoustic rules by performing critical band analysis, determining noise-like or tone-like areas of the audio signal, applying a thresholding rule to the signal, and then considering the absolute hearing threshold. Thereafter, the number of bits required to quantize the spectrum without introducing perceptible quantization errors is determined. For example, Painter & Spanias disclosed a formula for perceptual entropy in bits per second that is consistent with the ISO / IEC MPEG-1 psychoacoustic model 2 [Painter & Spanias, Perceptual Coding of Digital Audio, Proc. Of IEEE, Vol. 88, No. 4 (2000); also generally refer to the Moving Picture Experts Group standard https: / / mpeg.chiariglione.org / standards ; these two documents are incorporated herein by reference] are closely related.

[0085] Depending on the type of audio processing function applied, such as the multi-band dynamics processor described above, various optimization methods can maximize the PRI of an audio sample. For example, a sub-band dynamics compressor can be parameterized by a compression threshold, attack time, gain, and compression ratio for each f sub-band, and these parameters can be determined by an optimization process. In some cases, the effect of the multi-band dynamics processor on the audio signal is nonlinear and requires appropriate optimization techniques, such as gradient descent. For example, if the audio signal is processed in many sub-bands and multiple parameters need to be determined for each sub-band, the number of parameters that need to be determined may become larger. In such cases, it may not be feasible to optimize all parameters simultaneously, and a sequential approach can be used for parameter optimization. Although the sequential optimization process does not necessarily produce the best parameters, the parameter values ​​obtained will result in an increase in PRI relative to the unprocessed audio sample, thereby improving the listening experience of the listener.

[0086] Other parameterization processes known in the art may be used to calculate parameters based on user generated threshold and superthreshold information. For example, common canonical techniques for linear and nonlinear DSP may be employed. Well-known processes for linear hearing aid algorithms include POGO, NAL, and DSL. See, for example, H. Dillon, Hearing Aids, 2nd edition, Boomerang Press, 2012.

[0087] Fine tuning of any of the above techniques can be estimated from the manual fitting data. For example, it is common in the art to fit a multi-band dynamic processor based on a series of tests given to a patient, where the parameters are adjusted based on the patient's response, such as a series of A / B tests / decision tree paradigms, where the patient is asked which parameter set sounds better subjectively. This testing ultimately guides the optimal parameterization of the DSP. In an example of the present invention, the results of the manual fitting of DSP parameters for a given hearing curve can be sorted and entered into a server database. Subsequently, the results can be sorted based on the patient's response. Figure 6 and / or Figure 7 The method described in is used to calculate the user's parameters.

[0088] Fig.16 An example of a computing system 1600 is shown, which may be, for example, any computing device component (e.g., mobile device 100, server, etc.) or any component thereof, wherein the components of the system communicate with each other using connection 1605. Connection 1605 may be a physical connection via a bus, or a direct connection such as in a chipset architecture into processor 1610. Connection 1605 may also be a virtual connection, a networked connection, or a logical connection.

[0089] In some embodiments, computing system 1600 is a distributed system, where the functionality described in the present disclosure can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components, each of which performs some or all of the functionality for which the component is described. In some embodiments, the components can be physical or virtual devices.

[0090] Exemplary system 1600 includes at least one processing unit (CPU or processor) 1610 and connections 1605 that couple various system components including system memory 1615, such as read-only memory (ROM) 1620 and random access memory (RAM) 1625, to processor 1610. Computing system 1600 may include a cache of high-speed memory 1612 that is directly connected to, in close proximity to, or integrated as part of processor 1610.

[0091] Processor 1610 may include any general purpose processor and hardware or software services configured to control processor 1610 (such as services 1632, 1634, and 1636 stored in storage 1630), as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1610 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0092] To enable user interaction, computing system 1600 includes input device 1645, which can represent any number of input mechanisms, such as microphones for speech, touch-sensitive screens for gesture or graphic input, keyboards, mice, motion input, speech, etc. Computing system 1600 may also include output device 1635, which may be one or more of many output mechanisms known to those skilled in the art. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with computing system 1600. Computing system 1600 may include communication interface 1640, which may generally control and manage user input and system output. There is no restriction on operating on any particular hardware device, and therefore, the basic features herein may be easily replaced with improved hardware or firmware devices when developed.

[0093] The storage device 1630 may be a non-volatile memory device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM), a read-only memory (ROM), and / or some combination of these devices.

[0094] Storage 1630 may include software services, servers, services, etc., which cause the system to perform functions when the code defining such software is executed by processor 1610. In some embodiments, hardware services that perform a particular function may include a combination of software components stored in a computer-readable medium and the necessary hardware components (such as processor 1610, connection 1605, output device 1635, etc.) to perform the function.

[0095] The proposed technology provides an efficient and accurate way to personalize audio playback on multiple consumer electronic devices through server-adjusted sound personalization. It should be understood that the present disclosure encompasses many variations, options, and alternatives. For clarity of explanation, in some cases, the present technology may be represented as including individual functional blocks, which include functional blocks that include devices, device components, steps, or routines in methods embodied in software or a combination of hardware and software.

[0096] The method according to the above example can be implemented using computer executable instructions stored in or otherwise available from a computer readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions. The part of the computer resources used can be accessed through a network. The computer executable instructions can be, for example, a binary file, an intermediate format instruction (such as assembly language), firmware, or source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the example include a disk or optical disk, a flash memory, a USB device provided with a non-volatile memory, a networked storage device, and the like.

[0097] Devices implementing methods according to these disclosures may include hardware, firmware, and / or software, and may take any of a variety of form factors. Typical examples of such form factors include laptop computers, smart phones, small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, and the like. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By way of further example, such functionality may also be implemented between different chips or different processes executed in a single device on a circuit board. Instructions, media for communicating such instructions, computing resources for executing them, and other structures for supporting such computing resources are devices for providing the functionality described in these disclosures.

[0098] Although various examples and other information are used to explain various aspects within the scope of the appended claims, limitations on the claims should not be implied based on specific features or arrangements in such examples, as those of ordinary skill in the art will be able to use these examples to derive various implementations. In addition, although a subject matter may have been described in language specific to examples of structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to such features or actions. For example, such functionality may be distributed in different ways in components other than those identified herein or performed therein. More specifically, the features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

Claims

1. A method for processing an audio signal, the method comprising: performing a user hearing test using an audio personalization application running on the first audio output device; outputting user hearing data from the user hearing test to a server; storing the user hearing data on a database associated with the server, wherein the user hearing data is stored using a unique identifier of the user as a reference; Determining a set of calculated digital signal processing parameters for a given sound personalization algorithm based on the user hearing data; receiving, at the server, a request for personalized audio playback at a second audio output device, the request generated in response to an audio personalization application executing on the second audio output device receiving user input including a unique identifier of the user, wherein the second audio output device is different from the first audio output device; outputting the calculated digital signal processing parameters to the audio personalization application running on the second audio output device, wherein the calculated digital signal processing parameters are output by the server in response to receiving the request and the unique identifier of the user from the second audio output device; as well as An audio signal is processed on the second audio output device using the given sound personalization algorithm parameterized by the calculated digital signal processing parameters. 2 . The method of claim 1 , wherein determining the calculated set of digital signal processing parameters is performed remotely from one or more of the first audio output device and the second audio output device.

3. The method of claim 2, wherein the calculated set of digital signal processing parameters is determined on the server and the calculated digital signal processing parameters determined by the server are stored in the database using the unique identifier of the user as a reference.

4. The method of claim 1, wherein determining the calculated set of digital signal processing parameters is performed locally on one or more of the first audio output device and the second audio output device.

5. The method of claim 4, wherein the locally determined set of calculated digital signal processing parameters is stored in the database associated with the server using the unique identifier of the user as a reference.

6. The method of claim 1, wherein the calculated set of digital signal processing parameters is recalculated in response to a user performing an additional hearing test on any of a plurality of audio output devices including the first audio output device or the second audio output device.

7. The method of claim 6, wherein: The additional hearing test reflects information from another auditory filter; or the additional hearing test is a different type of hearing test to the hearing test previously performed; or The additional hearing test is an updated version of the hearing test previously performed.

8. The method of claim 6, wherein the additional hearing test replaces the hearing data from a previously conducted hearing test with an abnormal result.

9. The method of claim 8, wherein the parameter set of the parameterized given sound personalization algorithm comprises at least one of a threshold value of a dynamic range compressor provided in each sub-band, a ratio value of a dynamic range compressor provided in each sub-band, and a gain value provided in each sub-band.

10. The method of claim 8, wherein the given sound personalization algorithm is a multi-band dynamics processor.

11. The method of claim 1, wherein the hearing test is one or more of a threshold test, a suprathreshold test, a psychophysical tuning curve test, a masked threshold test, a temporal fine structure test, a speech hearing in noise test, and a temporal masking curve test.

12. The method of claim 1, wherein the hearing test measures masking threshold curves in a frequency range from 250 Hz to 12 kHz.

13. The method of claim 1, wherein the given sound personalization algorithm operates on sub-band signals of the audio signal.

14. The method of claim 1, wherein: determining the calculated set of digital signal processing parameters using a best fit of the user hearing data to previously input hearing data in a database of the server; or determining said calculated set of digital signal processing parameters using a fitting mathematical function derived from the plotted hearing and digital signal processing parameter data; and The parameters associated with the best fit and the previously input hearing data are selected to correspond to parameters of the user. The method of claim 14 , wherein the best fit is determined by one of an average Euclidean distance and a root mean square difference.

16. The method of claim 1, wherein the audio output device is one of a mobile phone, a tablet computer, a television, a laptop computer, a hearable device, a smart speaker, a headset, and a speaker system.

17. An audio processing system, comprising: at least one processor; as well as at least one memory storing instructions that, when executed, cause the at least one processor to perform actions comprising: performing a user hearing test using an audio personalization application running on the first audio output device; outputting user hearing data from the user hearing test to a server; storing the user hearing data on a database associated with the server, wherein the user hearing data is stored using a unique identifier of the user as a reference; Determining a set of calculated digital signal processing parameters for a given sound personalization algorithm based on the user hearing data; receiving, at the server, a request for personalized audio playback at a second audio output device, the request generated in response to an audio personalization application executing on the second audio output device receiving user input including a unique identifier of the user, wherein the second audio output device is different from the first audio output device; outputting the calculated digital signal processing parameters to the audio personalization application running on the second audio output device, wherein the calculated digital signal processing parameters are output by the server in response to receiving the request and the unique identifier of the user from the second audio output device; as well as An audio signal is processed on the second audio output device using the given sound personalization algorithm parameterized by the calculated digital signal processing parameters.

18. A non-transitory computer-readable storage medium storing a program containing instructions that, when executed on a processor of an audio output device, cause the processor to perform actions comprising: performing a user hearing test using an audio personalization application running on the first audio output device; outputting user hearing data from the user hearing test to a server; storing the user hearing data on a database associated with the server, wherein the user hearing data is stored using a unique identifier of the user as a reference; Determining a set of calculated digital signal processing parameters for a given sound personalization algorithm based on the user hearing data; receiving, at the server, a request for personalized audio playback at a second audio output device, the request generated in response to an audio personalization application executing on the second audio output device receiving user input including a unique identifier of the user, wherein the second audio output device is different from the first audio output device; outputting the calculated digital signal processing parameters to the audio personalization application running on the second audio output device, wherein the calculated digital signal processing parameters are output by the server in response to receiving the request and the unique identifier of the user at the second audio output device; as well as An audio signal is processed on the second audio output device using the given sound personalization algorithm parameterized by the calculated digital signal processing parameters.

Images