Volume leveler controller and controlling method

Abstract

To provide a volume leveler controller and a controlling method.SOLUTION: In one embodiment, a volume leveler controller includes an audio content classifier for identifying a content type of an audio signal in real time; and an adjusting unit for adjusting a volume leveler in a continuous manner on the basis of the identified content type. The adjusting unit may configured to positively correlate dynamic gain of the volume leveler with informative content types of the audio signal, and negatively correlate the dynamic gain of the volume leveler with interfering content types of the audio signal.SELECTED DRAWING: Figure 20

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to Chinese Patent Application No. 201310100422.1, filed March 26, 2013, and U.S. Provisional Patent Application No. 61 / 811,072, filed April 11, 2013, each of which is incorporated herein by reference in its entirety.

[0002] Technical Field FIELD OF THE DISCLOSURE This application relates generally to audio signal processing. In particular, embodiments of the present application relate to apparatus and methods for audio classification and processing, particularly for control of dialogue enhancers, surround virtualizers, volume levelers and equalizers. [Background technology]

[0003] Some audio improvement devices tend to modify the audio signal in either the time domain or the spectral domain in order to improve the overall quality of the audio and correspondingly enhance the user's experience. Various audio improvement devices have been developed for different purposes. Some typical examples of audio improvement devices include:

[0004] Dialogue Enhancer: Dialogue is the most important component for understanding the story in movies and radio or television programs. Methods of dialogue enhancement have been developed to increase the clarity and intelligibility of dialogue, especially for the elderly who are experiencing hearing loss.

[0005] Surround Virtualizer: The Surround Virtualizer allows surround (multi-channel) sound signals to be rendered through the PC's internal speakers or through headphones, i.e., using stereo equipment (such as speakers and headphones) to virtually create the effect of surround, providing a cinema experience for consumers.

[0006] Volume Leveler: The volume leveler aims to adjust the volume of audio content during playback so that the volume is fairly consistent over time based on a target loudness value.

[0007] Equalizer: An equalizer provides spectral balance consistency, known as "tone" or "timbre," and allows the user to configure the overall profile (curve or shape) of the frequency response (gain) in individual frequency bands to emphasize certain sounds or eliminate undesirable sounds. In a traditional equalizer, different equalizer presets may be provided for different sounds, such as different music genres. Once a preset is selected or an equalization profile is set, the same equalization gain is applied to the signal until the equalization profile is manually modified. In contrast, a dynamic equalizer achieves spectral balance consistency by continuously monitoring the spectral balance of the audio, comparing it to the desired tone, and dynamically adjusting the equalization filters to transform the original tone of the audio into the desired tone.

[0008] In general, an audio improvement device has its own application scenario / context. That is, an audio improvement device may only be suitable for a certain set of contents, and not for all possible audio signals, because different contents may need to be processed in different ways. For example, a dialogue enhancement method is usually applied to movie contents. If it were applied to music without dialogue, it may erroneously boost some frequency subbands, introducing severe timbre changes and perceptual inconsistencies. Similarly, if a noise suppression method were applied to a music signal, strong artifacts would be audible.

[0009] However, for an audio processing system, which typically includes a collection of audio improvement devices, the input can necessarily be any possible type of audio signal. For example, an audio processing system integrated into a PC will receive audio content from a variety of sources, including movies, music, VoIP, and games. Thus, it becomes important to identify or differentiate the content being processed in order to apply better algorithms or better parameters of each algorithm to the corresponding content.

[0010] In order to distinguish audio content and apply better parameters or better audio improvement algorithms accordingly, traditional systems typically pre-design a set of presets and require the user to select a preset for the content to be played. The preset typically encodes a set of audio improvement algorithms and / or their best parameters to be applied, such as a "Movie" preset and a "Music" preset specifically designed for movie or music playback. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] WO 2008 / 106036 (H. Muesch, “Speech Enhancement in Entertainment Audio”) [Patent Document 2] US Patent Application Publication No. 2009 / 0097676A1 (AJ Seefeldt et al., “Calculating and Adjusting the Perceived Loudness and / or the Perceived Spectral Balance of an Audio Signal”) [Patent Document 3] International Publication No. 2007 / 127023 (BG Grockett et al., "Audio Gain Control Using Specific-Loudness-Based Auditory Event Detection")

Patent document 4

Non-licensed literature

[0012]

Non-licensed literature 1

Non-licensed Document 2

Non-licensed Document 4

[0013] However, manual selection is inconvenient for users, who usually do not switch between predefined presets frequently, but simply stick with one preset for all content. Moreover, even in some automatic solutions, the parameters or algorithm setups in a preset are usually discrete (e.g., turn on or off a specific algorithm for a specific content), and do not allow adjusting the parameters in a content-based, continuous manner. [Means for solving the problem]

[0014] The first aspect of this application is to automatically configure the audio enhancement device in a continuous manner based on the audio content during playback. In this "automatic" mode, the user can simply enjoy the content without having to choose different presets. On the other hand, continuous adjustment is more important to avoid audible artifacts at the transition points.

[0015] According to an embodiment of the first aspect, the audio processing device includes an audio classifier for classifying an audio signal into at least one audio type in real time; an audio improvement device for improving a listener's experience; and an adjustment unit for adjusting at least one parameter of the audio improvement device in a continuous manner based on a confidence value of the at least one audio type.

[0016] The audio improvement device may be any of the following: a dialogue enhancer, a surround virtualizer, a volume leveller and an equalizer.

[0017] Correspondingly, an audio processing method includes classifying an audio signal into at least one audio type in real time; and adjusting at least one parameter for audio improvement in a continuous manner based on a confidence value of the at least one audio type.

[0018] According to another embodiment of the first aspect, the volume leveler controller includes an audio content classifier for identifying a content type of the audio signal in real time; and an adjustment unit for adjusting the volume leveler in a continuous manner based on the identified content type. The adjustment unit may be configured to positively correlate a dynamic gain of the volume leveler with an informative content type of the audio signal and to negatively correlate a dynamic gain of the volume leveler with an interfering content type of the audio signal.

[0019] An audio processing device having a volume leveller controller as above is also disclosed.

[0020] Correspondingly, the volume leveler control method includes identifying a content type of the audio signal in real time; and adjusting the volume leveler in a continuous manner based on the identified content type by positively correlating a dynamic gain of the volume leveler with an informative content type of the audio signal and negatively correlating a dynamic gain of the volume leveler with an interfering content type of the audio signal.

[0021] According to yet another embodiment of the first aspect, the equalizer controller includes an audio classifier for identifying an audio type of the audio signal in real time; and an adjustment unit for adjusting the equalizer in a continuous manner based on a confidence value of the identified audio type.

[0022] An audio processing apparatus having an equalizer controller as above is also disclosed.

[0023] Correspondingly, the equalizer control method includes identifying an audio type of an audio signal in real time; and adjusting the equalizer in a continuous manner based on a confidence value of the identified audio type.

[0024] The present application also provides a computer readable medium having computer program instructions recorded thereon which, when executed by a processor, cause the processor to perform the audio processing method or the volume leveller control method or the equalizer control method described above.

[0025] According to an embodiment of the first aspect, an audio improvement device, which may be one of a dialogue enhancer, a surround virtualizer, a volume leveller and an equalizer, may be continuously adjusted according to the type of audio signal and / or a confidence value for said type.

[0026] A second aspect of this application is to develop a content identification component that identifies multiple audio types. The detection results may be used to steer / guide the behavior of various audio enhancement devices in finding better parameters in a sequential manner.

[0027] According to an embodiment of the second aspect, the audio classifier includes: a short-term feature extractor configured to extract short-term features from short-term audio segments, each of which comprises a sequence of audio frames; a short-term classifier configured to classify a sequence of short-term segments in a long-term audio segment into short-term audio types using the respective short-term features; a statistics extractor configured to calculate statistics of results of the short-term classifier for the sequence of short-term segments in the long-term audio segment as long-term features; and a long-term classifier configured to classify the long-term audio segments into long-term audio types using the long-term features.

[0028] An audio processing device comprising an audio classifier as above is also disclosed.

[0029] Correspondingly, the audio classification method includes: extracting short-term features from short-term audio segments, each of which comprises a sequence of audio frames; classifying a sequence of short-term segments in a long-term audio segment into short-term audio types using the respective short-term features; calculating statistics of the results of the classification process for the sequence of short-term segments in the long-term audio segment as long-term features; and classifying the long-term audio segments into long-term audio types using the long-term features.

[0030] According to another embodiment of the second aspect, the audio classifier includes an audio content classifier that identifies a content type of a short-term segment of an audio signal; and an audio context classifier that identifies a context type of the short-term segment based at least in part on the content type identified by the audio content classifier.

[0031] An audio processing device comprising an audio classifier as above is also disclosed.

[0032] Correspondingly, the audio classification method includes identifying a content type of a short-term segment of an audio signal; and identifying a context type of said short-term segment based at least in part on the identified content type.

[0033] The present application also provides a computer readable medium having computer program instructions recorded thereon which, when executed by a processor, cause the processor to perform the audio classification method described above.

[0034] According to an embodiment of the second aspect, the audio signals may be classified into different long-term or context types that are distinct from the short-term or content types. The type of audio signal and / or its confidence value may further be used to adjust audio improvement devices such as a dialogue enhancer, a surround virtualizer, a volume leveller or an equalizer. [Brief description of the drawings]

[0035] The present application is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference symbols refer to similar elements and in which: [Figure 1] FIG. 1 illustrates an audio processing device according to an embodiment of the present application. [Diagram 2] FIG. 2 shows a modification of the embodiment shown in FIG. [Diagram 3] FIG. 2 shows a modification of the embodiment shown in FIG. [Figure 4] FIG. 2 illustrates a possible configuration of a classifier for identifying multiple audio types and calculating confidence values. [Diagram 5] FIG. 2 illustrates a possible configuration of a classifier for identifying multiple audio types and calculating confidence values. [Figure 6]FIG. 2 illustrates a possible configuration of a classifier for identifying multiple audio types and calculating confidence values. [Figure 7] 2 illustrates a further embodiment of the audio processing device of the present application; [Figure 8] 2 illustrates a further embodiment of the audio processing device of the present application; [Figure 9] 2 illustrates a further embodiment of the audio processing device of the present application; [Figure 10] FIG. 2 illustrates the delay in transitioning between various audio types. [Figure 11] 1 is a flow chart illustrating an audio processing method according to an embodiment of the present application; [Figure 12] 1 is a flow chart illustrating an audio processing method according to an embodiment of the present application; [Figure 13] 1 is a flow chart illustrating an audio processing method according to an embodiment of the present application; [Figure 14] 1 is a flow chart illustrating an audio processing method according to an embodiment of the present application; [Figure 15] FIG. 2 illustrates a dialog enhancer controller according to an embodiment of the present application. [Figure 16] 4 is a flow chart illustrating the use of the audio processing method according to the present application in controlling a dialogue enhancer. [Figure 17] 4 is a flow chart illustrating the use of the audio processing method according to the present application in controlling a dialogue enhancer. [Figure 18] FIG. 2 illustrates a surround virtualizer controller according to an embodiment of the present application. [Figure 19] 4 is a flow chart showing the use of the audio processing method according to the present application in controlling a surround virtualizer. [Figure 20] FIG. 2 illustrates a volume leveller controller according to an embodiment of the present application. [Figure 21] FIG. 13 illustrates the effect of a volume leveller controller according to the present application. [Figure 22]FIG. 2 illustrates an equalizer controller according to an embodiment of the present application. [Diagram 23] FIG. 13 shows some examples of desired spectral balance presets. [Figure 24] FIG. 2 illustrates an audio classifier according to an embodiment of the present application. [Diagram 25] FIG. 2 illustrates some features used by the audio classifier of the present application. [Figure 26] FIG. 2 illustrates some features used by the audio classifier of the present application. [Figure 27] FIG. 2 illustrates a further embodiment of an audio classifier according to the present application. [Figure 28] FIG. 2 illustrates a further embodiment of an audio classifier according to the present application. [Figure 29] FIG. 2 illustrates a further embodiment of an audio classifier according to the present application. [Diagram 30] 1 is a flow chart illustrating an audio classification method according to an embodiment of the present application; [Diagram 31] 1 is a flow chart illustrating an audio classification method according to an embodiment of the present application; [Diagram 32] 1 is a flow chart illustrating an audio classification method according to an embodiment of the present application; [Diagram 33] 1 is a flow chart illustrating an audio classification method according to an embodiment of the present application; [Diagram 34] FIG. 2 illustrates an audio classifier according to another embodiment of the present application. [Diagram 35] FIG. 2 illustrates an audio classifier according to yet another embodiment of the present application. [Diagram 36] FIG. 2 illustrates heuristic rules used in the audio classifier of the present application. [Figure 37] FIG. 2 illustrates a further embodiment of an audio classifier according to the present application. [Figure 38] FIG. 2 illustrates a further embodiment of an audio classifier according to the present application. [Figure 39]1 is a flow chart illustrating an audio classification method according to an embodiment of the present application; [Diagram 40] 1 is a flow chart illustrating an audio classification method according to an embodiment of the present application; [Diagram 41] FIG. 1 is a block diagram illustrating an exemplary system for implementing embodiments of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] In the following, the embodiments of the present application will be described with reference to the drawings. It should be noted that for clarity, the representations and descriptions of components and processes known to those skilled in the art but not necessary for understanding the present application will be omitted in the drawings and descriptions.

[0037] As will be appreciated by one skilled in the art, aspects of the present application may be embodied as a system, an apparatus (e.g., a mobile phone, a portable media player, a personal computer, a browser, a television set-top box or a digital video recorder or any other media player), a method, or a computer program product. Thus, aspects of the present application may take the form of a hardware embodiment, a software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining both software and hardware aspects, all of which may be referred to herein as a "circuit," a "module," or a "system." Furthermore, aspects of the present application may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied therein.

[0038] Any combination of one or more computer readable media may be utilized. A computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device or any suitable combination of the above. Further specific examples of computer readable storage media (not an exhaustive list) include: an electrical connection with one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device or any suitable combination of the above. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus or device.

[0039] A computer-readable signal medium may include a propagated data signal in which computer-readable program code is embodied, for example in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic or optical signals, or any suitable combination thereof.

[0040] A computer-readable signal medium may be any computer-readable medium, other than a computer-readable storage medium, that can communicate, propagate or carry a program for use by or in connection with an instruction execution system, apparatus or device.

[0041] The program code embodied on the computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the above.

[0042] Computer program code for carrying out operations of aspects of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and the like, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may run entirely on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In this last scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider).

[0043] Aspects of the present invention will be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to generate a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the function / step specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0044] These computer program instructions may be stored on a computer-readable medium that can direct a computer, other programmable data processing apparatus or other device to function in a particular manner, such that the instructions stored on the computer-readable medium create an article of manufacture that includes instructions that implement the functions / steps identified in one or more blocks of the flowcharts and / or block diagrams.

[0045] The computer program instructions may be loaded into a computer, other programmable data processing apparatus or other device and cause the computer, other programmable apparatus or other device to perform a series of operations to create a computer-implemented process such that the instructions executing on the computer or other programmable apparatus provide a process for implementing the functions / steps identified in one or more blocks of the flowcharts and / or block diagrams.

[0046] In the following, embodiments of the present application are described in detail. For clarity, the description is organized into the following sections: Part 1: Audio processing apparatus and methods Section 1.1 Audio type Section 1.2 Audio-based confidence scores and classifier construction Section 1.3 Audio-type confidence smoothing Section 1.4 Parameter Adjustment Section 1.5 Parameter Smoothing Section 1.6 Audio Type Transitions Section 1.7 Combinations of embodiments and application scenarios Section 1.8 Audio Processing Methods Part II: Dialogue Enhancer Controller and Control Method Section 2.1 Levels of dialogue improvement Section 2.2 Thresholds for determining frequency bands to be improved Section 2.3 Background Level Adjustment Section 2.4 Combinations of embodiments and application scenarios Section 2.5 Dialogue Enhancer Control Method Part 3: Surround Virtualizer Controller and Control Method Section 3.1 Surround Boost Amount Section 3.2 Starting Frequency Section 3.3 Combinations of embodiments and application scenarios Section 3.4 Surround virtualizer control method Part 4: Volume leveller controller and control method Section 4.1 Informative and Interactive Content Types Section 4.2 Content types in different contexts Section 4.3 Context Types Section 4.4 Combinations of embodiments and application scenarios Section 4.5 Volume Leveler Control Method Part 5: Equalizer controller and control method Section 5.1 Content-type-based control Section 5.2 Predominant source likelihood in music Section 5.3 Equalizer Presets Section 5.4 Context-based control Section 5.5 Combinations of embodiments and application scenarios Section 5.6 Equalizer control method Part VI: Audio classifiers and classification methods Section 6.1 Context Classifier Based on Content Type Classification Section 6.2 Extracting Long-Term Features Section 6.3 Extracting Short-Term Features Section 6.4 Combinations of embodiments and application scenarios Section 6.5 Audio Classification Methods Part 7: VoIP classifiers and classification methods Section 7.1 Context Classification Based on Short-Term Segments Section 7.2 Classification using VoIP speech and VoIP noise Section 7.3 Smoothing Fluctuations Section 7.4 Combinations of embodiments and application scenarios Section 7.5 VoIP classification method.

[0047] Part 1: Audio processing device and method 1 shows a general framework of a content-adaptive audio processing device 100 that supports automatic configuration of at least one audio enhancement device with enhanced parameters based on the audio content during playback. It has three main components: an audio classifier 200, an adjustment unit 300 and an audio enhancement device 400.

[0048] The audio classifier 200 classifies an audio signal into at least one audio type in real time. It automatically identifies the audio type of the content during playback. Any audio classification technique can be applied to identify the audio content, such as through signal processing, machine learning, and pattern recognition. A confidence value representing the probability of the audio content with respect to a predefined set of target audio types is estimated almost simultaneously.

[0049] The audio enhancement device 400 enhances the audience experience by performing processing on the audio signal, and is discussed in more detail below.

[0050] The adjustment unit 300 is for adjusting at least one parameter of the audio improvement device in a continuous manner based on a confidence value of the at least one audio type. It is designed to steer the behavior of the audio improvement device 400. It estimates the most suitable parameters of the corresponding audio improvement device based on the results obtained from the audio classifier 200.

[0051] Various audio improvement devices can be applied in this system. Figure 2 shows an exemplary system including four audio improvement devices, including a Dialog Enhancer (DE) 402, a Surround Virtualizer (SV) 404, a Volume Leveler (VL) 406 and an Equalizer (EQ) 408. Each audio improvement device can be automatically adjusted in a continuous manner based on the results (audio type and / or confidence value) obtained in the audio classifier 200.

[0052] Of course, the audio processing device may not necessarily include all types of audio improvement devices, but may only include one or more of them. On the other hand, the audio improvement devices are not limited to the devices given in this disclosure, but may include additional types of audio improvement devices, which are also within the scope of this application. Furthermore, the names of the audio improvement devices discussed in this disclosure, including the dialogue enhancer (DE) 402, the surround virtualizer (SV) 404, the volume leveler (VL) 406, and the equalizer (EQ) 408, are not intended to be limiting, and each of them is to be construed to cover any other device that achieves the same or similar functions.

[0053] (Section 1.1 Audio Type) In order to properly control various kinds of audio enhancement devices, the present application further provides a new configuration of audio types, although the audio types in the prior art are also applicable in the present application.

[0054] Specifically, audio types from different semantic levels are modeled, including low-level audio elements that represent fundamental components in an audio signal, and high-level audio genres that represent the most common audio content in real-world user entertainment applications. The former may be referred to as "content types". Basic audio content types may include speech, music (including songs), background sounds (or sound effects), and noise.

[0055] The meaning of speech and music is clear. Noise in this application means physical noise, not semantic noise. Physical noise in this application may include noise caused by technical reasons, such as noise from an air conditioner or pink noise caused by the signal transmission path. In contrast, "background sounds" in this application are sound effects that may be auditory events occurring around the core target of the listener's attention. For example, in the audio signal of a telephone call, besides the voice of the speaker, there may be some other sounds that are not intended, such as the voice of some other person not related to the call, keyboard sounds, footsteps, etc. These unwanted sounds are called "background sounds" rather than noise. In other words, "background sounds" may be defined as sounds that are not the target (or the core target of the listener's attention) or even are unwanted, but still have some content meaning. On the other hand, "noise" may be defined as unwanted sounds excluding the target on and background sounds.

[0056] Sometimes, background sounds are not really "unwanted", but are intentionally generated and carry some useful information, such as background sounds in movies, television programs, or radio broadcast programs. Thus, they are sometimes referred to as "sound effects". In the remainder of this disclosure, for the sake of brevity, only "background sounds" will be used, and may also be shortened to "background".

[0057] Furthermore, music may be further classified into music without dominant sources and music with dominant sources. If there is a source (voice or instrument) that is much stronger than the other sources in a piece of music, it is referred to as "music with dominant sources". If not, it is referred to as "music without dominant sources". For example, polyphonic music with vocals and various instruments is considered to be music without dominant sources if it is harmonically balanced or the energy of the several most prominent sources is comparable to each other. In contrast, if one source (e.g. voice) is much louder while the other sources are much quieter, it is considered to contain dominant sources. As another example, prominent or prominent instrumental tones are "music with dominant sources".

[0058] Music may further be classified into different types based on various criteria. Music may be classified based on musical genres such as, but not limited to, rock, jazz, rap, and folk. Music may also be classified based on instruments, such as vocal and instrumental music. Instrumental music may include a variety of music played using different instruments, such as piano music and guitar music. Other exemplary criteria include rhythm, tempo, musical timbre, and / or any other musical attribute by which music may be grouped based on similarity of attributes. For example, based on timbre, vocal music may be classified into tenor, baritone, bass, soprano, mezzo-soprano, and alto.

[0059] The content type of an audio signal may be classified with respect to short-term audio segments that consist of multiple frames. Typically, an audio frame has a duration of multiple milliseconds, such as 20 ms, and the length of a short-term audio segment to be classified by an audio classifier may have a duration of hundreds of milliseconds to a few seconds, e.g., 1 second.

[0060] To control the audio improvement device in a content-adaptive manner, the audio signal may be classified in real-time. For the above content types, the content type of the current short-term audio segment represents the content type of the current audio signal. Since the length of the short-term audio segments is not very long, the audio signal may be divided as sequential, non-overlapping short-term audio segments. However, the short-term audio segments may be sampled continuously / semi-continuously along the time axis of the audio signal. That is, the short-term audio segments may be sampled using a window with a predetermined length (the intended length of the short-term audio segment) moving with a step size of one or more frames along the time axis of the audio signal.

[0061] High level audio genres are sometimes referred to as "contextual types" because they indicate the long term type of an audio signal and may be considered as the environment or context of a current sound event, which may be classified into content types as above. According to this application, contextual types may include most common audio applications such as cinematic media, music (including songs), games, and VoIP (Voice over Internet Protocol).

[0062] The meaning of music, games and VoIP is self-explanatory. Cinematic media may include movies, television programs, radio broadcast programs or any other audio media similar to those mentioned above. The main characteristic of cinematic media is the possible mixture of speech, music and various kinds of background sounds (sound effects).

[0063] It may be noted that both the content type and the context type include music (including songs), and hereinafter the terms "short-term music" and "long-term music" are used to distinguish between them, respectively.

[0064] For some embodiments of the present application, some other context-based configurations are also proposed.

[0065] For example, audio signals may be classified as high quality audio (such as cinematic media and music CDs) or low quality audio (such as VoIP, low bitrate online streaming audio and user generated content), which may be collectively referred to as "audio quality types."

[0066] As another example, an audio signal may be classified as VoIP or non-VoIP. These may be considered as transformations of the four context type configurations mentioned above (VoIP, cinematic media, (long-term) music and gaming). In the context of a VoIP or non-VoIP context, an audio signal may be classified as a VoIP-related audio content type, such as VoIP speech, non-VoIP speech, VoIP noise and non-VoIP noise. The VoIP audio content type configuration is particularly useful for distinguishing between VoIP and non-VoIP contexts, since the VoIP context is usually the most challenging application scenario for a volume leveler (a type of audio improvement device).

[0067] In general, a context type of an audio signal may be classified with respect to a long-term audio segment that is longer than a short-term audio segment. A long-term audio segment may consist of a number of frames that is greater than the number of frames in a short-term audio segment. A long-term audio segment may consist of a number of short-term audio segments. In general, a long-term audio segment may have a length on the order of seconds, from a few seconds to tens of seconds, e.g., 10 seconds.

[0068] Similarly, the audio signal may be classified into context types in real time in order to control the audio improvement device in an adaptive manner. Similarly, the context type of the current long-term audio segment represents the context type of the current audio signal. Since the length of the long-term audio segment is relatively long, the audio signal may be sampled continuously / semi-continuously along the time axis of the audio signal in order to avoid abrupt changes of its context type and thus of the operating parameters of the audio improvement device(s). That is, the long-term audio segment may be sampled using a window with a predetermined length (the intended length of the long-term audio segment) running along the time axis of the audio signal with a step size of one or more frames or one or more short-term segments.

[0069] Above, both content types and context types have been described. In an embodiment of the present application, the adjustment unit 300 may adjust at least one parameter of the audio improvement device(s) based on at least one of the various content types and / or at least one of the various context types. Thus, as shown in Fig. 3, in a variation of the embodiment shown in Fig. 1, the audio classifier 200 may comprise an audio content classifier 202 or an audio context classifier 204 or both.

[0070] In the above, different audio types based on different standards (e.g. for context types) and different audio types at different hierarchical levels (e.g. for content types) are mentioned. However, these standards and hierarchical levels are merely for convenience of description herein and are in no way limiting. That is, in the present application, any two or more of the above-mentioned audio types can be identified by the audio classifier 200 and considered by the adjustment unit 300 at the same time, as will be described later. That is, all audio types at different hierarchical levels may be parallel or at the same level.

[0071] Section 1.2 Audio-based confidence values ​​and classifier construction The audio classifier 200 may output a hard decision result, or the adjustment unit 300 may consider the result of the audio classifier 200 as a hard decision result. Even for hard decisions, multiple audio types can be assigned to an audio segment. For example, an audio segment can be a mixed signal of speech and short-term music, and therefore can be labeled with both "speech" and "short-term music". The resulting labels can be directly used to steer the audio improvement device(s) 400. A simple example is to enable the dialogue enhancer 402 when speech is present and turn it off when speech is not present. However, this hard decision method may introduce some artificiality at the transition point from one audio type to another without a careful smoothing scheme (described below).

[0072] To adjust the parameters of the audio enhancer in a more flexible and continuous manner, a confidence value for each target audio type can be estimated (soft decision). The confidence value represents the level of match between the audio content to be identified and the target audio type, with a value between 0 and 1.

[0073] As mentioned earlier, many classification techniques may directly output a confidence value. Confidence values ​​can also be calculated from various methods that may be considered part of the classifier. For example, if an audio model is trained by some probabilistic modeling technique such as Gaussian Mixture Models (GMM), then the posterior probability can be used to represent the confidence value as follows:

[0074]

number

[0075] On the other hand, when audio models are trained using some discriminative methods such as Support Vector Machine (SVM) and adaBoost, only a score (real-valued) is obtained from the model comparison. In these cases, the following sigmoid function is usually used to map the obtained score (theoretically -∞ to ∞) to an expected confidence (0 to 1):

[0076]

number

[0077] For some embodiments of the present application, the conditioning unit 300 may use more than two content types and / or more than two context types, in which case the audio content classifier 202 needs to identify more than two content types and / or the audio context classifier 204 needs to identify more than two context types. In such a situation, the audio content classifier 202 or the audio context classifier 204 may be a group of classifiers organized in a certain configuration.

[0078] For example, if the adjustment unit 300 requires all four context types, cinematic media, long-term music, gaming and VoIP, the audio context classifier 204 may have the following different configurations:

[0079] First, the audio context classifier 204 may have six one-vs-one binary classifiers (each classifier discriminating one target audio type from another target audio type) organized as shown in FIG. 4, three one-vs-other binary classifiers (each classifier discriminating one target audio type from other audio types) organized as shown in FIG. 5, and four one-vs-other classifiers organized as shown in FIG. 6. There are other configurations, such as a Decision Directed Acyclic Graph (DDAG) configuration. Note that in FIGS. 4-6 and the corresponding description below, "movie" is used instead of "cinematic media" for brevity.

[0080] Each binary classifier gives a confidence score H(x) for its output, where x represents an audio segment. After the outputs of each binary classifier are obtained, they need to be mapped to final confidence values ​​for the identified context types.

[0081] In general, an audio signal is classified into M context types, where M is a positive integer. A normal one-to-one configuration builds M(M-1) / 2 classifiers, where each classifier is trained with data from two classes. Then, each one-to-one classifier casts a vote for its preferred class, and the final result is the class with the most votes among the classifications of the M(M-1) / 2 classifiers. Compared with the normal one-to-one configuration, the hierarchical configuration in Fig. 4 also requires building M(M-1) / 2 classifiers. However, since the segment x is judged to be / not be in the corresponding class at each hierarchical level, and the overall level count is M-1, the test iteration process can be shortened to M-1. The final confidence values ​​for the various context types can be calculated using, for example, the binary classification confidence value H k (x), where k=1,2,...,6 represent different context types.

[0082]

number

[0083]

number

[0084]

number

[0085] It should be noted that in the configurations shown in Figures 4-6, the sequences of the various binary classifiers are not necessarily as shown, but may be other sequences, which may be selected by manual assignment or automatic learning according to different requirements of different applications.

[0086] The above description is directed to the audio context classifier 204. For the audio content classifier 202, the situation is similar.

[0087] Alternatively, the audio content classifier 202 or the audio context classifier 204 may be implemented as one single classifier that simultaneously identifies all content / context types and simultaneously gives corresponding confidence values. There are many existing techniques to do this.

[0088] Using the confidence values, the output of the audio classifier 200 can be represented as a vector. Each dimension of the vector represents a confidence value for each target audio type. For example, if the target audio types are sequential (speech, short-term music, noise, background), an exemplary output result can be (0.9, 0.5, 0.0, 0.0), indicating that we are 90% certain that the audio content is speech and 50% certain that the audio is music. Note that not all dimensions in the output vector need to sum to 1 (e.g., the results from FIG. 6 are not necessarily normalized). That is, the audio signal may be a mixture of speech and short-term music.

[0089] Later, in Parts 6 and 7, we will discuss in detail the novel implementation of audio context classification and audio content classification.

[0090] <Section 1.3 Audio-type confidence smoothing> Optionally, after each audio segment has been classified into a predefined audio type, an additional step is to smooth the classification result along the time axis in order to avoid abrupt jumps from one type to another and to allow a smoother estimation of the parameters in the audio enhancement device. For example, if a long excerpt is classified as cinematic media except for one segment classified as VoIP, the abrupt VoIP decision can be corrected to cinematic media by smoothing.

[0091] Therefore, in a variant of the embodiment shown in FIG. 7, a type smoothing unit 712 is further provided to smooth, for each audio type, the confidence value of the audio signal at the current time.

[0092] A common smoothing method is based on a weighted average, such as calculating a weighted sum of the current actual confidence value and the last smoothed confidence value.

[0093] smoothConf(t)=β·smoothConf(t-1)+(1-β)·conf(t) (3) where t is the current time (current audio segment), t-1 is the last time (last audio segment), β is the weight, and conf and smoothConf are the confidence values ​​before and after smoothing, respectively.

[0094] In terms of confidence values, the results from the hard decisions of the classifier can also be represented using confidence values ​​whose values ​​are either 0 or 1. That is, if a target audio type is chosen and assigned to the audio segment, the corresponding confidence value is 1, otherwise the confidence value is 0. Thus, even if the audio classifier 200 does not provide confidence values ​​but only provides hard decisions regarding the audio type, continuous adjustment of the adjustment unit 300 is still possible through the smoothing operation of the type smoothing unit 712.

[0095] The smoothing algorithm can be "asymmetric" by using different smoothing weights for different cases. For example, the weights for computing the weighted sum may be adaptively changed based on the confidence value of the audio type of the audio signal. The higher the confidence value of the current segment, the higher its weight.

[0096] From another perspective, the weights for computing the weighted sum may be adaptively varied based on different pairs of transitions from one audio type to another, especially when the audio improvement device(s) is adjusted based on multiple content types identified by the audio classifier 200, rather than based on the presence or absence of one single content type alone. For example, for a transition from an audio type that appears more frequently in a context to another audio type that appears less frequently in that context, the confidence value of the latter may be smoothed so that it does not increase too quickly, since it may be a random outburst.

[0097] Another factor is the trend of change (increasing or decreasing), including the rate of change. If we say we care more about delays when a certain audio type becomes present (i.e., when its confidence value increases), we can design a smoothing algorithm as follows:

[0098]

number

[0099] From a different perspective, considering the change trend of a certain audio type is just a particular example of considering different transition pairs of audio types. For example, increasing the confidence value of type A may be considered as a transition from non-A to A, and decreasing the confidence value of type A may be considered as a transition from A to non-A.

[0100] (Section 1.4 Parameter Adjustment) The adjustment unit 300 is designed to estimate or adjust the appropriate parameters for the audio improvement device(s) 400 based on the obtained results from the audio classifier 200. Different adjustment algorithms may be designed for different audio improvement devices using either content type or context type or both for joint judgment. For example, for context type information such as cinematic media and long-term music, presets as described above can be automatically selected and applied to the corresponding content. With the available content type information, the parameters of each audio improvement device can be adjusted in a finer manner, as will be shown in the following section. The content type information and the context information can further be used jointly in the adjustment unit 300 to balance the long-term and short-term information. A specific adjustment algorithm for a specific audio improvement device may be considered as a separate adjustment unit. Alternatively, different adjustment algorithms may be considered together as a combined adjustment unit.

[0101] That is, the adjustment unit 300 may be configured to adjust the at least one parameter of the audio improvement device based on the confidence value of at least one content type and / or the confidence value of at least one context type. For a particular audio improvement device, some of the audio types are informative and some of the audio types are interfering. Thus, the parameters of a particular audio improvement device may be positively or negatively correlated with the confidence value(s) of the informative audio type(s) or the interfering audio type(s). Here, "positively correlated" means that the parameter increases or decreases linearly or non-linearly with an increase or decrease in the confidence value of the audio type. "Negatively correlated" means that the parameter increases or decreases linearly or non-linearly, respectively, with a decrease or increase in the confidence value of the audio type.

[0102] Here, the decrease and increase of the confidence value is "transferred" directly to the parameter to be adjusted by positive or negative correlation. In mathematics, such correlation or "transfer" may be embodied as a direct or inverse proportionality, a plus or minus (addition or subtraction) operation, a multiplication or division or a non-linear function. All these forms of correlation may be referred to as a "transfer function". To determine the increase or decrease of the confidence value, the current confidence value or a mathematical transformation thereof may also be compared to the last confidence value or a number of historical confidence values ​​or a mathematical transformation thereof. In the context of this application, the term "comparison" means a comparison through subtraction or a comparison through division. The increase or decrease can be determined by determining whether the difference is greater than 0 or whether the ratio is greater than 1.

[0103] In particular implementations, the parameters can be directly related to the confidence values ​​or their ratios or differences through appropriate algorithms (such as transfer functions) without the need for an "external observer" to explicitly know that a particular confidence value and / or a particular parameter has increased or decreased. Some particular examples are given in Parts 2-5 below for particular audio improvement devices.

[0104] As mentioned in the previous section, for the same audio segment, the classifier 200 may identify multiple audio types with respective confidence values. Since an audio segment may contain multiple components at the same time, such as music, speech and background sounds, their confidence values ​​may not necessarily sum to one. In such a situation, the parameters of the audio improvement device need to be balanced between the different audio types. For example, the adjustment unit 300 may be configured to take into account at least some of the multiple audio types through weighting the confidence value of the at least one audio type based on the importance of the at least one audio type. The more important a certain audio type is, the more the parameters are influenced by it.

[0105] The weights may also reflect the informative and coherent effects of the audio types. For example, coherent audio types may be given a negative weight. Some specific examples are given in Parts 2-5 below for specific audio enhancement devices.

[0106] Note that in the context of this application, "weight" has a broader meaning than a coefficient in a polynomial. In addition to a coefficient in a polynomial, a "weight" can also take the form of an exponent or power. When a coefficient in a polynomial, the weighting coefficients may or may not be normalized. In a nutshell, a weight simply represents how much influence the weighted object has on the parameter to be adjusted.

[0107] In some other embodiments, for multiple audio types included in the same audio segment, the confidence values ​​may be converted into weights through normalization. Then, the final parameters may be determined through calculating the sum of parameter preset values ​​predefined for each audio type and weighted by the weights based on the confidence values. That is, the adjustment unit 300 may be configured to take multiple audio types into account through weighting the effects of the multiple audio types based on the confidence values.

[0108] As a particular example of weighting, the adjustment unit is configured to consider at least one dominant audio type based on a confidence value. Audio types with too low (below a threshold) confidence value may not be considered. This is equivalent to the weights of other audio types with confidence values ​​below the threshold being set to 0. Some particular examples are given below in parts 2-5 of the particular audio improving device.

[0109] The content type and the context type can be considered together. In one embodiment, they can be considered to be at the same level and their confidence values ​​can have respective weights. In another embodiment, as the name suggests, a "context type" is a context or environment in which the "context type" is located, and thus the adjustment unit 200 may be configured such that content types in an audio signal of different context types are assigned different weights depending on the context type of the audio signal. In general, any audio type can constitute a context for another audio type, and the adjustment unit 200 may be configured to modify the weight of one audio type with the confidence value of another audio type. Some specific examples are given in the following parts II to V of specific audio improvement devices.

[0110] In the context of this application, the term "parameter" has a broader meaning than its literal meaning. In addition to parameters with a single value, parameters may refer to a set of parameters, a vector or profile of parameters, and presets as described above. In particular, but not limited to, the following parameters are discussed in Parts 2-5 below: level of dialogue enhancement, threshold for determining frequency bands to be dialogue enhanced, background level, amount of surround boost, start frequency for a surround virtualizer, dynamic gain or dynamic gain range of a volume leveler, a parameter indicating the degree to which an audio signal is a new perceptible audio event, equalization level, equalization profile, and spectral balance presets.

[0111] <Section 1.5 Parameter Smoothing> In section 1.3, we discussed smoothing the confidence values ​​of the audio type to avoid abrupt changes and thus avoid abrupt changes in the parameters of the audio enhancement device. Other measures are possible. One is to smooth the parameters that are adjusted based on the audio type, which is discussed in this section. Another is to configure the audio classifier and / or the adjustment unit to slow down changes in the results of the audio classifier, which is discussed in section 1.6.

[0112] In one embodiment, the parameters can be further smoothed to avoid fast changes that can introduce audible artifacts at the transition points, as follows:

[0113]

number

[0114] That is, as shown in Fig. 8, the audio processing device may comprise a parameter smoothing unit 814, which smoothes, for parameters of the audio improvement devices (such as at least one of the dialogue enhancer 402, the surround virtualizer 404, the volume leveller 406 and the equalizer 408) adjusted by the adjustment unit 300, the parameter values ​​determined by the adjustment unit 300 at the current time by calculating a weighted sum of the parameter values ​​determined by the adjustment unit at the current time and the smoothed parameter values ​​at the last time.

[0115] The time constant τ can be a fixed value based on the particular requirements of the application and / or implementation of the audio improvement apparatus 400. The time constant τ may be adaptively changed based on the audio type, in particular based on different transition types from one audio type to another, such as music to speech, speech to music, etc.

[0116] Take an equalizer as an example (more details can be mentioned in Part 5). Equalization is good to apply to music content, but bad to apply to speech content. Thus, in order to smooth the level of equalization, the time constant may be relatively small when the audio signal transitions from music to speech, so that a smaller equalization level can be applied more quickly to the speech content. On the other hand, the time constant for the transition from speech to music can be relatively large to avoid audible artifacts at the transition point.

[0117] To estimate the transition type (e.g. speech to music or music to speech), the content classification results can be used directly. That is, once the audio content is classified as music or speech, obtaining the transition type becomes simple. Instead of directly comparing the hard decision of the audio type, one can also rely on the estimated unsmoothed equalization level, so that the transition can be estimated in a more continuous way. The general idea is that if the unsmoothed equalization level increases, it indicates a transition from speech to music (or more musical), otherwise it is closer to a transition from music to speech (or more speech). By distinguishing between different transition types, the time constants can be set correspondingly. An example is as follows:

[0118]

number

[0119] (Section 1.6 Audio Type Transitions) 9 and 10, another scheme for avoiding abrupt changes in audio type and therefore in the parameters of an audio enhancement device is described.

[0120] 9, the audio processing device 100 may further comprise a timer 916 for measuring the duration for which the audio classifier 200 continuously outputs the same new audio type. The adjustment unit 300 may be configured to continue using the current audio type until the length of the duration of the new audio type reaches a threshold value.

[0121] In other words, an observation (or maintenance) phase is introduced, as shown in Fig. 10. In the observation phase (corresponding to a duration length threshold) the change in audio type is further monitored over a continuous amount of time to establish whether the audio type has really changed, before the adjustment unit 300 actually uses the new audio type.

[0122] As shown in FIG. 10, arrow (1) indicates the situation where the current state is type A and the result of the audio classifier 200 does not change.

[0123] If the current state is type A and the result of the audio classifier 200 is type B, the timer 916 starts timing, or as shown in Fig. 10, the process enters the observation phase (arrow (2)) and an initial value of the hangover count cnt is set, which indicates the length of the observation duration (equal to the threshold value).

[0124] Then, the audio classifier 200 continuously outputs type B, cnt continuously decreases (arrow (3)), and finally when cnt becomes equal to 0 (i.e. the duration length of the new type B reaches a threshold), the adjustment unit 300 may use the new audio type B (arrow (4)). Or, in other words, only at this point in time may the audio type be considered to have really changed to type B.

[0125] Otherwise, if the output of the audio classifier 200 reverts to the original type A before cnt becomes 0 (before the duration length reaches the threshold), the observation phase is terminated and the training unit 300 continues to use the original type A (arrow (5)).

[0126] The transformation from type B to type A may be similar to the process described above.

[0127] In the above process, the threshold (or remaining count) may be set based on the requirements of the application. It may be a predefined fixed value. It may be set adaptively. In one variation, the threshold is different for different transition pairs from one audio type to another. For example, when changing from type A to type B, the threshold may be a first value, and when changing from type B to type A, the threshold may be a second value.

[0128] In another variation, the hangover count (threshold) may be negatively correlated with the confidence value of the new audio type. The general idea is that if the confidence value indicates confusion between the two types (e.g., when the confidence value is only around 0.5), the observation duration needs to be long. Otherwise, the duration may be relatively short. Following this guideline, an exemplary hangover count can be set by the following formula:

[0129] HangCnt=C·|0.5-Conf|+D where HangCnt is the remaining duration or threshold, and C and D are two parameters that can be set based on application requirements, where C is usually negative and D is a positive value.

[0130] It is noted that the timer 916 (and thus the transition process described above) has been described as part of the audio processing device, but external to the audio classifier 200. In some other embodiments, it may be considered part of the audio classifier 200, as described in Section 7.3.

[0131] <Section 1.7 Combinations of embodiments and application scenarios> All of the embodiments and variations thereof discussed above may be implemented in any combination thereof, and any components mentioned in different parts / embodiments but having the same or similar functionality may be implemented as the same or separate components.

[0132] Specifically, when describing the embodiments and variants thereof above, components with similar reference numbers as those already described in the previous embodiment or variant are omitted, and only different configurations are described. In fact, these different components can be combined with components of other embodiments or variants, or alone to form separate solutions. For example, any two or more of the solutions described with reference to Figures 1 to 10 may be combined with each other. As the most complete solution, the audio processing device may have both the audio content classifier 202 and the audio context classifier 204, as well as the smoothing unit 712, the parameter smoothing unit 814 and the timer 916.

[0133] As mentioned above, the audio improvement device 400 may include a dialogue enhancer 402, a surround virtualizer 404, a volume leveler 406 and an equalizer 408. The audio processing device 100 may include any one or more of them, and the adjustment unit 300 may be adapted thereto. When involving multiple audio improvement devices 400, the adjustment unit 300 may be considered as including multiple sub-units 300A-300D (FIGS. 15, 18, 20 and 22) specific to each audio improvement device 400, or may still be considered as one combined adjustment unit. When specific to an audio improvement device, the adjustment unit 300 together with the audio classifier 200 and other possible components may be considered as a controller for that particular audio improvement device. This will be discussed in more detail in Parts II to V below.

[0134] Furthermore, the audio enhancement device 400 is not limited to the above examples and may include any other audio enhancement device.

[0135] Furthermore, any of the solutions already discussed or any combination thereof may be further combined with any of the embodiments described or implied in other parts of this disclosure, in particular the audio classifier embodiments discussed in Parts 6 and 7 may be used in an audio processing device.

[0136] (Section 1.8 Audio Processing Method) It is clear that in the course of describing the audio processing device in the above embodiments, some processes or methods are also disclosed. In the following, an overview of these methods is given, without repeating some of the details already discussed above. However, although these methods are disclosed in the course of describing the audio processing device, these methods do not necessarily employ or are not necessarily executed by the components described. For example, the embodiments of the audio processing device may be partially or fully realized using hardware and / or firmware, while the audio processing methods discussed below may employ the hardware and / or firmware of the audio processing device, but may be fully realized by a computer-executable program.

[0137] These methods are described below with reference to Figures 11 to 14. It should be noted that when the methods are implemented in real time, various operations are repeated in response to streaming properties of the audio signal, and the different operations are not necessarily with respect to the same audio segment.

[0138] In the embodiment shown in Fig. 11, an audio processing method is provided. First, an audio signal to be processed is classified into at least one audio type in real time (operation 1102). Based on a confidence value of the at least one audio type, at least one parameter for audio improvement can be continuously adjusted (operation 1104). The audio improvement may be dialogue improvement (operation 1106), surround virtualization (operation 1108), volume leveling (operation 1110) and / or equalization (operation 1112). Correspondingly, the at least one parameter may include at least one parameter for at least one of dialogue improvement processing, surround virtualization processing, volume leveling processing and equalization processing.

[0139] Here, "real-time" and "continuously" mean that the audio type, and therefore the parameters, change in real-time with the particular content of the audio signal. "Continuously" also means that the adjustments are not abrupt or discrete, but rather are continuous adjustments based on the confidence values.

[0140] The audio type may include a content type and / or a context type. Correspondingly, the adjusting operation 1104 may be configured to adjust the at least one parameter based on the confidence value of the at least one content type and the confidence value of the at least one context type. The content type may further include at least one of short-term music, speech, background sound and noise content types. The context type may further include at least one of long-term music, cinematic media, gaming and VoIP context types.

[0141] Several other context type schemes are also proposed, such as a VoIP-related context type, which includes VoIP and non-VoIP, and an audio quality type, which includes high quality audio or low quality audio.

[0142] The short-term music may be further classified into subtypes according to various criteria. It may include music without dominant sources and music with dominant sources depending on the presence of dominant sources. Furthermore, the short-term music may include at least one genre-based cluster or at least one instrument-based cluster or at least one music cluster classified based on the rhythm, tempo, timbre and / or any other musical attribute of the music.

[0143] When both content type and context type are identified, the importance of a content type may be determined by the context type in which it is located. That is, content types in an audio signal of different context types are assigned different weights depending on the context type of the audio signal. More generally, one audio type may influence another audio type or may be a prerequisite for another audio type. Thus, the adjustment 1104 operation may be configured to modify the weight of one audio type using the confidence value of another audio type.

[0144] When an audio signal is classified into multiple audio types simultaneously (i.e. for the same audio segment), the Adjustment 1104 operation may take into account some or all of the identified audio types for adjusting the parameter(s) for improving the audio segment. For example, the Adjustment 1104 operation may be configured to weight a confidence value of the at least one audio type based on the importance of the at least one audio type. Alternatively, the Adjustment 1104 operation may be configured to take into account at least some of the audio types through weighting them based on their confidence values. In one particular case, the Adjustment 1104 operation may be configured to take into account at least one dominant audio type based on a confidence value.

[0145] To avoid sudden changes in the results, a smoothing scheme may be introduced.

[0146] The adjusted parameter values ​​may be smoothed (operation 1214 of FIG. 12). For example, the parameter value determined by the adjustment 1104 operation at the current time may be replaced by a weighted sum of the parameter value determined by the adjustment operation at the current time and the smoothed parameter value at the last time. In this manner, through successively repeated smoothing operations, the parameter values ​​are smoothed over time.

[0147] The weights for calculating the weighted sum may be adaptively changed based on the audio type of the audio signal or based on different transition pairs from one audio type to another audio type, or alternatively, the weights for calculating the weighted sum are adaptively changed based on an increasing or decreasing trend of the parameter value determined by the adjustment operation.

[0148] Another smoothing scheme is shown in Fig. 13. That is, the method may further include smoothing the confidence values ​​of the audio signal at the current time by calculating, for each audio type, a weighted sum of the actual confidence value at the current time and the smoothed confidence value at the last time (operation 1303). Similar to the parameter smoothing operation 1214, the weights for calculating the weighted sum may be adaptively changed based on the confidence values ​​of the audio types of the audio signal or based on different transition pairs from one audio type to another audio type.

[0149] Another smoothing scheme is a buffer mechanism that delays the transition from one audio type to another even if the output of the audio classification operation 1102 changes, i.e., the adjustment 1104 operation waits for the output of the audio classification operation 1102 to stabilize rather than immediately using the new audio type.

[0150] Specifically, the method may include measuring the duration that the classification operation continuously outputs the same new audio type (operation 1403 in FIG. 14). Here, the adjustment 1104 operation is configured to continue using the current audio type ("N" in operation 14035 and operation 11041) until the duration length of the new audio type reaches a certain threshold ("Y" in operation 14035 and operation 11042). Specifically, when the audio type output from the audio classification operation 1102 changes with respect to the current audio type used in the audio parameter adjustment operation 1104 ("Y" in operation 1403), timing begins (operation 14032). If the audio classification operation 1102 continues to output new audio types, i.e., the determination in operation 14031 remains "Y", timing continues (operation 14032). Finally, when the duration of the new audio type reaches the threshold ("Y" in operation 14035), the adjustment operation 1104 uses the new audio type (operation 11042) and the timer is reset for the next switch of audio type (operation 14034). Until the threshold is reached ("N" in operation 14035), the adjustment operation 1104 continues to use the current audio type (operation 11041).

[0151] Here, the timing may be implemented using a timer mechanism (counting up or down). If after the timing has started but before the threshold is reached, the output of the audio classification operation 1102 reverts to the current audio type used in the adjustment operation 1104, it should be considered that there is no change with respect to the current audio type used by the adjustment operation 1104 ("N" in operation 14031). However, the current classification result (corresponding to the current audio segment to be classified in the audio signal) has changed with respect to the previous output of the audio classification operation 1102 (corresponding to the previous audio segment to be classified in the audio signal) ("Y" in operation 14033), so the timing is reset (operation 14034) until the next change ("Y" in operation 14031) starts the timing. Of course, if the classification result of the audio classification operation 1102 does not change with respect to the current audio type used by the audio parameter adjustment operation 1104 ("N" in operation 14031) and also does not change with respect to the previous classification ("N" in operation 14033), this indicates that the audio classification is in a stable state and the current audio type continues to be used.

[0152] The threshold used here may be different for different transition pairs from one audio type to another, since when a state is not very stable, it may generally be preferable for the audio improvement device to be in its default state rather than other states. On the other hand, it is safer to transition to a new audio type if the confidence value of the new audio type is relatively high. Therefore, the threshold may be negatively correlated with the confidence value of the new audio type: the higher the confidence value, the lower the threshold, i.e., the faster the audio type may transition to a new audio type.

[0153] As with the audio processing device embodiments, any combination of the audio processing method embodiments and variants thereof is feasible. On the other hand, any aspect of the audio processing method embodiments and variants thereof may be a separate solution. In particular, in all of the audio processing methods, audio classification methods as discussed in Parts 6 and 7 may be used.

[0154] Part 2: Dialogue enhancer controller and control method One example of an audio improvement device is the Dialogue Enhancer (DE), which aims to intermittently monitor the audio during playback, detect the presence of dialogue, and enhance the dialogue to increase its clarity and intelligibility (making it easier to hear and understand), especially for elderly people with declining hearing. In addition to detecting whether dialogue is present, it also detects the frequencies most important for intelligibility if dialogue is present and thus enhances accordingly (using dynamic spectral rebalancing). An exemplary dialogue enhancement method is presented in US Pat. No. 6,399,433, the entirety of which is incorporated herein by reference.

[0155] A common manual configuration setting for dialogue enhancers is that they are typically enabled for cinematic media content but disabled for music content, as dialogue enhancement can be too prone to false triggering for music signals.

[0156] With the available audio type information, the level of dialogue enhancement and other parameters can be adjusted based on the confidence value of the identified audio type. As a specific example of the audio processing device and method discussed above, the dialogue enhancer may use all the embodiments discussed in the first part and any combination of those embodiments. In particular, when controlling the dialogue enhancer, the audio classifier 200 and the adjustment unit 300 in the audio processing device 100 as shown in Figs. 1-10 may constitute a dialogue enhancer controller 1500 as shown in Fig. 15. In this embodiment, the adjustment unit is specific to the dialogue enhancer and may be referred to as 300A. As discussed in the previous part, the audio classifier 200 may include at least one of the audio content classifier 202 and the audio context classifier 204, and the dialogue enhancer controller 1500 may further include at least one of the type smoothing unit 712, the parameter smoothing unit 814 and the timer 916.

[0157] Therefore, in this section we will not repeat what has already been described in previous sections, but will simply provide some specific examples of this section.

[0158] For the dialogue enhancer, adjustable parameters include, but are not limited to, the level of dialogue enhancement, the background level, and a threshold for determining the frequency band to be enhanced. See U.S. Patent No. 6,333,636, the entire contents of which are incorporated herein by reference.

[0159] <Section 2.1 Levels of dialogue improvement> When it concerns the level of dialogue enhancement, the adjustment unit 300A may be configured to positively correlate the level of dialogue enhancement of the dialogue enhancer with the confidence value of the speech. Additionally or alternatively, the level may be negatively correlated with the confidence value of other content types. Thus, the level of dialogue enhancement can be set to be proportional (linearly or non-linearly) to the speech confidence. Thus, dialogue enhancement is less effective for non-speech signals such as music and background sounds (sound effects).

[0160] For the context type, the adjustment unit 300A may be configured to positively correlate the level of dialogue enhancement of the dialogue enhancer with the confidence value of cinematic media and / or VoIP, and negatively correlate the level of dialogue enhancement of the dialogue enhancer with the confidence value of long-term music and / or games. For example, the level of dialogue enhancement can be set to be proportional (linearly or non-linearly) to the confidence value of cinematic media. When the cinematic media confidence value is 0 (e.g., in music content), the level of dialogue enhancement is also 0, which is equivalent to disabling the dialogue enhancer.

[0161] As mentioned in the previous section, content types and context types may be considered jointly.

[0162] <Section 2.2 Thresholds for determining frequency bands to be improved> During operation of the dialogue enhancer, for each frequency band there is a threshold (usually an energy or loudness threshold) to determine whether it needs to be enhanced. That is, frequency bands above the respective energy / loudness threshold are enhanced. To adjust those thresholds, the adjustment unit 300A may be configured to positively correlate the threshold with the short-term music and / or noise and / or background sound confidence value and / or negatively correlate the threshold with the speech confidence value. For example, if the speech confidence is high, the threshold can be lowered assuming more reliable speech detection, allowing more frequency bands to be enhanced. On the other hand, if the music confidence value is high, the threshold can be higher, allowing fewer frequency bands to be enhanced (thus resulting in less artifacts).

[0163] (Section 2.3 Background Level Adjustment) Another component in the dialogue enhancer is the minimum tracking unit 4022 as shown in FIG. 15. It is used to estimate the background level in the audio signal (for SNR estimation and frequency band threshold estimation as described in section 2.2). It can also be adjusted based on the confidence value of the audio content type. For example, if the speech confidence value is high, the minimum tracking unit can be more confident to set the background level to the current minimum. If the music confidence is high, the background level can be set a little higher than its current minimum, or alternatively to a weighted average of the current minimum and the energy of the current frame, with a large weight on the current minimum. If the noise and background confidence is high, the background level can be set much higher than the current minimum, or alternatively to a weighted average of the current minimum and the energy of the current frame, with a small weight on the current minimum.

[0164] Thus, the adjustment unit 300A may be configured to assign an adjustment to the background level estimated by the minimum tracking unit, where the adjustment unit is further configured to positively correlate the adjustment with the short-term music and / or noise and / or background sound confidence value and / or to negatively correlate the adjustment with the speech confidence value. In one variation, the adjustment unit 300A may be configured to correlate the adjustment more positively with the noise and / or background sound confidence value than with the short-term music.

[0165] <Section 2.4 Combinations of embodiments and application scenarios> As with Part 1, all of the embodiments and variations thereof discussed above may be implemented in any combination thereof, and any components mentioned in different parts / embodiments but having the same or similar functionality may be implemented as the same or separate components.

[0166] For example, any two or more of the solutions described in Sections 2.1 to 2.3 may be combined with each other. These combinations may then be further combined with any of the embodiments described or implied in Part 1 and other parts below. In particular, many formulas are actually applicable to each type of audio improvement device or method, but they are not necessarily described or discussed in each part of this disclosure. In such cases, cross-references may be made between parts of this disclosure to apply a particular formula discussed in one part to another part with only appropriate adjustments of the relevant parameters, coefficients, powers and weights according to the specific requirements of a particular application.

[0167] <Section 2.5 Dialogue Enhancer Control Method> As in the first part, in the course of describing the dialogue enhancer controller in the above embodiment, it is clear that several processes or methods are also disclosed. In the following, an overview of these methods is given without repeating some of the details already discussed above.

[0168] First, the embodiment of the audio processing method discussed in Part 1 may be used for a dialogue enhancer, where the parameter(s) of the dialogue enhancer are one of the targets to be adjusted by the audio processing method. From this point of view, the audio processing method is also a dialogue enhancer control method.

[0169] In this section only the specific aspects of controlling the dialogue enhancer are discussed, for general aspects of the control method reference can be made to Part 1.

[0170] According to an embodiment, the audio processing method may further include a dialogue enhancement process, and the adjustment 1104 operation includes positively correlating the level of dialogue enhancement with a cinematic media and / or VoIP confidence value and / or negatively correlating the level of dialogue enhancement with a long-term music and / or gaming confidence value, i.e., the dialogue enhancement is primarily directed to audio signals in the cinematic media or VoIP context.

[0171] More specifically, the act of adjusting 1104 may include positively correlating the level of dialogue enhancement of the dialogue enhancer with the confidence value of the utterance.

[0172] The present application may adjust frequency bands to be enhanced in a dialogue enhancement process. As shown in Fig. 16, a threshold (usually energy or loudness) for determining whether each frequency band should be enhanced may be adjusted (operation 1602) based on the confidence value(s) of the identified audio type according to the present application. Then, within the dialogue enhancer, based on the adjusted threshold, frequency bands above each threshold are selected (operation 1604) and enhanced (operation 1606).

[0173] Specifically, the act of adjusting 1104 may include positively correlating those thresholds with the confidence value of short-term music and / or noise and / or background sounds and / or negatively correlating those thresholds with the confidence value of speech.

[0174] The audio processing method (particularly the dialogue enhancement processing) generally further comprises estimating a background level in the audio signal. This is generally implemented by a minimum tracking unit 4022 realized in the dialogue enhancer 402, and is used in the SNR estimation or the frequency band threshold estimation. The present application may be used to adjust the background level. In such a situation, after the background level is estimated (operation 1702), the background level is first adjusted based on the confidence value(s) of the audio type(s) (operation 1704) and then used in the SNR estimation and / or the frequency band threshold estimation (operation 1706). In particular, the adjustment 1104 operation may be configured to assign an adjustment to the estimated background level. Here, the adjustment 1104 operation may be configured to positively correlate the adjustment with the confidence value of short-term music and / or noise and / or background sounds and / or to negatively correlate the adjustment with the confidence value of speech.

[0175] More specifically, the Adjust 1104 operation may be configured to make the adjustment more positively correlated with the confidence value of noise and / or background than short-term music.

[0176] As with the audio processing device embodiments, any combination of the audio processing method embodiments and variants thereof is feasible. On the other hand, any aspect of the audio processing method embodiments and variants thereof may be separate solutions. Furthermore, any two or more of the solutions described in this section may be combined with each other, and these combinations may further be combined with any embodiment described or implied in the first part or other parts below.

[0177] Part 3: Surround virtualizer controller and control method Surround virtualizers allow surround sound signals (such as multi-channel 5.1 and 7.1) to be rendered through the PC's internal speakers or through headphones, i.e. using stereo equipment such as built-in laptop speakers or headphones to virtually create a surround effect and provide a cinema experience for consumers. Surround virtualizers typically use Head Related Transfer Functions (HRTFs) to simulate the arrival of sounds at the ears from the various speaker positions associated with the multi-channel audio signal.

[0178] Current surround virtualizers work well on headphones, but they work differently with built-in speakers and for different content: general medical, cinematic media content will enable the surround virtualizer for speakers, whereas music will not, as it can sound too thin.

[0179] Since the same parameters in the surround virtualizer cannot create a good sound image for both cinematic media and music content at the same time, the parameters need to be more precisely adjusted based on the content. With the available audio type information, especially the music confidence value and the speech confidence value, as well as any other content type and context information, the functionality can be performed with the present application.

[0180] As with the second part, as a specific example of the audio processing device and method discussed in the first part, the surround virtualizer 404 may use all the embodiments discussed in the first part and any combination of those embodiments disclosed therein. In particular, when controlling the surround virtualizer, the audio classifier 200 and the adjustment unit 300 in the audio processing device 100 as shown in Figs. 1-10 may constitute a surround virtualizer controller 1800 as shown in Fig. 18. In this embodiment, the adjustment unit is specific to the surround virtualizer, so it may be referred to as 300B. As with the second part, the audio classifier 200 may include at least one of the audio content classifier 202 and the audio context classifier 204, and the surround virtualizer controller 1800 may further include at least one of the type smoothing unit 712, the parameter smoothing unit 814, and the timer 916.

[0181] Therefore, in this section we will not repeat what has already been described in the first section, but will simply provide some specific examples of this section.

[0182] For the Surround Virtualizer, adjustable parameters include, but are not limited to, the amount of surround boost and the start frequency of the Surround Virtualizer 404.

[0183] (Section 3.1 Surround Boost Amount) When concerned with the amount of surround boost, the adjustment unit 300B may be configured to positively correlate the amount of surround boost of the surround virtualizer 404 with the noise and / or background and / or speech confidence value and / or to negatively correlate the amount of surround boost with the short-term music confidence value.

[0184] In particular, to modify the surround virtualizer 404 so that music (content type) sounds acceptable, an exemplary implementation of the adjustment unit 300B can adjust the amount of surround boost based on the short-term music confidence value. SB ∝ (1-Conf music ) (5) where SB is the amount of surround boost and Conf music is the short-term music confidence value.

[0185] It reduces the surround boost on the music, preventing it from sounding lifeless.

[0186] Similarly, utterance confidence values ​​can be used, e.g. SB ∝ (1-Conf music )*Conf speech α (6) Here, Conf speech is the confidence value of the speech, and α is a weighting coefficient in the form of an exponent, which may range from 1 to 2. This formula indicates that the amount of surround boost will be high only for pure speech (high speech confidence and low music confidence).

[0187] Alternatively, we can consider only the confidence value of the utterance. SB∝Conf speech (7) Various variations can be designed in a similar manner. In particular, for noise or background sounds, formulas similar to formulas (5) to (7) may be constructed. Furthermore, the effects of the four content types may be considered together in any combination. In such a situation, noise and background are ambient sounds and are safer to have a large boost amount. Speech may have a medium boost amount, assuming that the speaker usually sits in front of the screen. Thus, the adjustment unit 300B may be configured to make the surround boost amount more positively correlated with the confidence value of noise and / or background than that of content type speech.

[0188] Another alternative can be applied, assuming we pre-define the expected boost amount for each content type (which is equated with the weight).

[0189]

number

[0190] From another perspective, α with the subscript of a content type in formula (8) is the expected / predefined boost amount of that content type, and the quotient of the trust value of the corresponding content type divided by the sum of the trust values ​​of all identified content types may be regarded as a normalized weight of the predefined / expected boost amount of the corresponding content type. That is, the adjusting unit 300B may be configured to take into account at least some of the multiple content types through weighting the predefined boost amounts of the multiple content types based on their trust values.

[0191] For the context type, the adjustment unit 300B may be configured to positively correlate the amount of surround boost of the surround virtualizer 404 with the confidence value for cinematic media and / or games, and negatively correlate the amount of surround boost with the confidence value for long-term music and / or VoIP. Then, formulas similar to (5) through (8) may be constructed.

[0192] As a specific example, the surround virtualizer 404 can be enabled for pure cinematic media and / or games, and disabled for music and / or VoIP. Meanwhile, the boost amount of the surround virtualizer 404 can be set differently for cinematic media and games, with cinematic media using a higher boost amount and games using a lower one. Thus, the adjustment unit 300B may be configured to make the surround boost amount more positively correlated with the confidence value for cinematic media than for games.

[0193] Similar to the content types, the amount of boost for the audio signal can also be set to a weighted average of the confidence values ​​of the context types.

[0194]

number

[0195] Also similar to the content types, the subscripted α of a context type in formula (9) is the expected / predefined boost amount of that context type, and the quotient of the confidence value of the corresponding context type divided by the sum of the confidence values ​​of all identified context types may be regarded as a normalized weight of the predefined / expected boost amount of the corresponding context type. That is, the adjusting unit 300B may be configured to take into account at least some of the multiple context types through weighting the predefined boost amounts of the multiple context types based on their confidence values.

[0196] (Section 3.2 Starting Frequency) Other parameters can also be modified in the surround virtualizer, such as the start frequency. In general, higher frequency components in an audio signal are more suitable to be rendered spatially. For example, in music, the bass sounds strange if it is rendered with more surround effect. Thus, for a particular audio signal, the surround virtualizer needs to determine a frequency threshold above which the components are rendered spatially and below which the components are preserved. The frequency threshold is the start frequency.

[0197] According to an embodiment of the present application, the start frequency for the surround virtualizer can be increased for music content, so that more base can be preserved for the music signal. The adjustment unit 300B can then be configured to positively correlate the start frequency of the surround virtualizer with the short-term music confidence value.

[0198] <Section 3.3 Combinations of embodiments and application scenarios> As with Part 1, all of the embodiments and variations thereof discussed above may be implemented in any combination thereof, and any components mentioned in different parts / embodiments but having the same or similar functionality may be implemented as the same or separate components.

[0199] For example, any two or more of the solutions described in Sections 3.1 and 3.2 may be combined with each other, and any of these combinations may be further combined with any of the embodiments described or implied in Parts 1, 2 and other parts below.

[0200] <Section 3.4 Surround virtualizer control method> As in the first part, in the course of describing the surround virtualizer controller in the above embodiment, it is clear that several processes or methods are also disclosed. In the following, an overview of these methods is given without repeating some of the details already discussed above.

[0201] First, the embodiment of the audio processing method discussed in the first part may be used for a surround virtualizer, where the parameter(s) of the surround virtualizer are one of the targets to be adjusted by the audio processing method, from this point of view the audio processing method is also a surround virtualizer control method.

[0202] In this section only the aspects specific to the control of the surround virtualizer are discussed, for general aspects of the control method reference can be made to part 1.

[0203] According to an embodiment, the audio processing method may further include a surround virtualization process, and the adjusting operation 1104 may be configured to positively correlate a surround boost amount of the surround virtualization process with the noise and / or background and / or speech confidence value and / or to negatively correlate a surround boost amount with the short-term music confidence value.

[0204] In particular, the adjusting operation 1104 may be configured to make the amount of surround boost more positively correlated with the confidence value of noise and / or background and / or speech than with the content-type speech.

[0205] Alternatively or additionally, the amount of surround boost may be adjusted based on the confidence value(s) of the context type(s). In particular, the adjusting operation 1104 may be configured to positively correlate the amount of surround boost of the surround virtualization process with a confidence value for cinematic media and / or games and / or negatively correlate the amount of surround boost with a confidence value for long-term music and / or VoIP.

[0206] In particular, the adjusting operation 1104 may be configured to make the amount of surround boost more positively correlated with cinematic media than with games.

[0207] Another parameter to be adjusted is the start frequency for the surround virtualization process. As shown in Fig. 19, the start frequency is first adjusted based on the confidence value(s) of the audio type(s) (operation 1902), and then the surround virtualizer processes the audio components above the start frequency (operation 1904). In particular, the adjusting operation 1104 may be configured to positively correlate the start frequency of the surround virtualization process with the confidence value of short-term music.

[0208] As with the audio processing device embodiments, any combination of the audio processing method embodiments and variations thereof is practical. On the other hand, any aspect of the audio processing method embodiments and variations thereof may be separate solutions. Furthermore, any two or more of the solutions described in this section may be combined with each other, and these combinations may further be combined with any embodiment described or implied in other parts of this disclosure.

[0209] Part 4: Volume leveller controller and control method The volumes of different audio sources or different pieces of the same audio source sometimes vary widely. This is irritating as the user has to adjust the volume frequently. A Volume Leveler (VL) aims to adjust the volume of audio content during playback so that it is mostly consistent over time based on a target loudness value. Exemplary volume levelers are described in U.S. Patent Nos. 5,393,663, 5,431,362, 5,441,233, and 5,563,471, all of which are incorporated herein by reference in their entirety.

[0210] A volume leveler continuously measures the loudness of an audio signal in some way and then modifies the signal by an amount of gain. The gain is a scaling factor for modifying the loudness of an audio signal and is usually a function of the measured loudness, the desired target loudness and some other factors. To estimate the proper gain several factors need to be considered, along with the underlying criteria for approaching the target loudness and maintaining the dynamic range. It usually includes several sub-elements such as automatic gain control (AGC), auditory event detection and dynamic range control (DRC).

[0211] Control signals are commonly applied in volume levelers to control the "gain" of an audio signal. For example, the control signal can be an indicator of a change in the loudness of the audio signal, derived by pure signal analysis. The control signal can also be an audio event indicator, which indicates whether a new audio event appears, through psychoacoustic analysis, such as auditory scene analysis or specific-loudness-based auditory event detection. Such control signals are applied in volume levelers for gain control, for example, by ensuring that the gain is approximately constant within an auditory event to reduce possible audible artifacts due to abrupt changes in gain in the audio signal, and by constraining many of the gain changes to the vicinity of event boundaries.

[0212] However, conventional methods of deriving control signals cannot distinguish informative auditory events from non-informative (interfering) auditory events. Here, informative auditory events represent audio events that contain meaningful information, such as dialogue and music, and may be paid more attention by users. On the other hand, non-informative signals do not contain meaningful information for users, such as noise in VoIP. As a result, non-informative signals may also be applied with a large gain and boosted to near the target loudness, which may be unpleasant in some applications. For example, in VoIP calls, noise signals that appear in pauses in speech are often boosted to a large volume after being processed by a volume leveler, which is not desired by users.

[0213] To address this problem at least partially, the present application proposes to control a volume leveller based on the embodiments discussed in the first part.

[0214] As a specific example of the audio processing device and method discussed in Part 1, as in Parts 2 and 3, the volume leveler 406 may use all the embodiments discussed in Part 1 and any combination of those embodiments disclosed therein. In particular, when controlling the volume leveler 406, the audio classifier 200 and the adjustment unit 300 in the audio processing device 100 as shown in Figures 1-10 may constitute a volume leveler 406 controller 2000 as shown in Figure 20. In this embodiment, the adjustment unit is specific to the volume leveler 406 and may be referred to as 300C.

[0215] That is, based on the disclosure in the first part, the volume leveller controller 2000 may have an audio classifier 200 for continuously identifying an audio type (such as a content type and / or a context type) of an audio signal, and an adjustment unit 300C for adjusting the volume leveller in a continuous manner based on a confidence value of the identified audio type. Similarly, the audio classifier 200 may include at least one of an audio content classifier 202 and an audio context classifier 204, and the volume leveller controller 2000 may further include at least one of a type smoothing unit 712, a parameter smoothing unit 814, and a timer 916.

[0216] Therefore, in this section we will not repeat what has already been described in the first section, but will simply provide some specific examples of this section.

[0217] Based on the classification results, various parameters of the volume leveller 406 can be adaptively adjusted. A parameter directly related to the dynamic gain or the range of the dynamic gain can be adjusted, for example by reducing the gain for non-informative signals. A parameter indicating the degree to which the signal is a new perceptible audio event can also be adjusted and the dynamic gain can be indirectly controlled (the gain can change slowly within an audio event, but abruptly at the border of two audio events). Several embodiments of parameter adjustment or volume leveller control mechanisms are presented in this application.

[0218] <Section 4.1 Informative and Interference Content Types> As discussed above, with respect to volume leveler control, audio content types can be categorized as informational content types and interferential content types. The adjustment unit 300C may be configured to positively correlate the dynamic gain of the volume leveller with the informational content type of the audio signal and to negatively correlate the dynamic gain of the volume leveller with the coherent content type of the audio signal.

[0219] As an example, noise is coherent (non-informational) and becomes annoying when boosted to high volumes. Parameters that directly control dynamic gain or indicate new audio events are: GainControl∝1-Conf noise (10) As shown, the noise confidence value (Conf noise ) can be set to be proportional to a decreasing function of

[0220] Here, for simplicity, the symbol GainControl is used to represent all parameters (or their effects) related to gain control in the volume leveler, since different implementations of the volume leveler may use different names for the parameters with different underlying meanings. Using the single term GainControl allows for a shorter representation without loss of generality. In essence, adjusting these parameters is equivalent to applying linear or nonlinear weights to the original gain. As an example, GainControl can be used directly to scale the gain, such that if GainControl is small, the gain is small. As another specific example, the gain is indirectly controlled by scaling an event control signal with GainControl as described in US Pat. No. 6,399,363, which is incorporated herein by reference in its entirety. In this case, when GainControl is small, the control of the gain of the volume leveler is modified to prevent the gain from changing significantly over time. When GainControl is large, the control is modified to allow the gain of the leveler to change more freely.

[0221] Using the gain control described in formula (10) (directly scaling the original gain or event control signal), the dynamic gain of an audio signal is correlated (linearly or nonlinearly) to its noise confidence value. If a signal is noisy with high confidence value, the dynamic gain is scaled by a factor (1-Conf noise), thus avoiding boosting the noise signal to an uncomfortably large volume.

[0222] As a variation of formula (10), if the application (such as VoIP) does not care about background noise, the background noise can be treated similarly and applied with a small gain. The control function is the noise confidence value (Conf noise ) and background confidence value (Conf bkg ) can be taken into account. For example, GainControl∝(1-Conf noise )·(1-Conf bkg ) (11) In the above formula, since both noise and background sounds are unwanted, GainControl is equally affected by the noise confidence value and the background confidence value. This can be considered as noise and background sounds having the same weight. Depending on the situation, they may have different weights. For example, different coefficients or different exponents (α and γ) may be given to the noise and background sound confidence values ​​(or their difference from 1). That is, formula (11) becomes GainControl∝(1-Conf noise ) α ·(1-Conf bkg ) γ (12) or GainControl∝(1-Conf noise α )·(1-Conf bkg γ ) (13) It may be rewritten as:

[0223] Alternatively, the adjustment unit 300C may be configured to take into account at least one prevalent content type based on the trust value. GainControl∝1-max(Conf noise ,Conf bkg ) (14) Both formula (11) (and its variants) and formula (14) show small gains for noise and background signals, and the original behavior of the volume leveler only holds when both the noise and background beliefs are small and GainControl is close to 1 (as in speech and music signals).

[0224] The above example considers a predominant interfering content type. Depending on the circumstances, the adjustment unit 300C may be configured to consider a predominant informational content type based on the confidence value. More generally, the adjustment unit 300C may be configured to consider at least one predominant content type based on the confidence value, regardless of whether the identified audio type is / includes an informational and / or an interfering audio type.

[0225] As another exemplary variation of formula (10), assuming that the speech signal has the most informative content and requires fewer modifications to the default behavior of the volume leveller, the control function can be the noise confidence value (Conf noise ) and utterance confidence value (Conf speech ) GainControl∝1-Conf noise ·(1-Conf speech ) (15) With this number, a small GainControl is obtained only for signals with high noise confidence and low speech confidence (e.g. pure noise), while for high speech confidence GainControl is close to 1 (thus preserving the original behavior of the volume leveller). More generally, for a certain content type (Conf noise The weight of at least one other content type (Conf speech In formula (15) above, the confidence of the utterance can be considered as changing the weighting factor of the confidence of the noise (a different kind of weight compared to the weights in formulas (12) and (13)). In other words, in formula (10), Conf noisecan be considered as 1, while in formula (15), some other audio type (such as but not limited to speech) affects the importance of the noise confidence value. Thus, Conf noise It can be said that the weights of are modified by the confidence value of the utterance. In the context of the present disclosure, the term "weight" is interpreted to include this, i.e., to indicate the importance of the value, but not necessarily normalized. Section 1.4 may be referenced.

[0226] From another perspective, similar to formulas (12) and (13), weights in the form of exponentials can be applied to the confidence values ​​in the above functions to indicate the priority (or importance) of different audio signals. For example, formula (15) can be modified as follows:

[0227] GainControl∝1-Conf noise α ·(1-Conf speech ) γ (16) where α and γ are two weights, which can be set smaller if a larger response to modifying the leveler parameters is expected.

[0228] Formulas (10)-(16) can be freely combined to form various control functions that may be suitable for different applications. Confidence values ​​of other audio content types, such as music confidence values, can also be easily incorporated into the control functions in a similar manner.

[0229] In the case where GainControl is used to adjust a parameter that indicates the degree to which the signal is a new perceptible audio event and thus indirectly control the dynamic gain (the gain changes slowly within an audio event but can change rapidly at the border between two audio events), one may consider there to be another transfer function between the content type confidence value and the final dynamic gain.

[0230] Section 4.2 Content Types in Various Contexts The above control functions in formulas (10)-(16) take into account the confidence values ​​of audio content types such as noise, background sounds, short-term music and speech, but do not consider the audio context from which the sounds come, such as cinematic media and VoIP. The same audio content type may need to be treated differently in different audio contexts, for example background sounds. Background sounds include a variety of sounds such as car engines, explosions and applause. What may not be meaningful in VoIP may be important in cinematic media. This indicates that the audio contexts of interest need to be identified and different control functions designed for different audio contexts.

[0231] Thus, the adjustment unit 300C may be configured to consider the content type of the audio signal as informative or interfering based on the context type of the audio signal. For example, by considering a noise confidence value and a background confidence value and distinguishing between VoIP and non-VoIP contexts, the audio context dependent control function can be:

[0232] if audio context is VoIP GainControl∝1-max(Conf noise ,Conf bkg ) else GainControl∝1-Conf noise (17) That is, in a VoIP context, noise and background sounds are considered as interfering content types, whereas in a non-VoIP context, background sounds are considered as an informational content type.

[0233] As another example, an audio context dependent control function that considers confidence values ​​for speech, noise and background and distinguishes between VoIP and non-VoIP contexts can be as follows:

[0234] if audio context is VoIP GainControl∝1-max(Conf noise ,Conf bkg ) else GainControl∝1-Conf noise ·(1-Conf speech ) (18) Here, speech is emphasized as an informational content type.

[0235] If music is also important information in non-VoIP contexts, the second half of formula (18) can be written as GainControl∝1-Conf noise ·(1-max(Conf speech ,Conf music ) (19) It can be expanded as follows.

[0236] Indeed, each of the control functions in (10)-(16) or variations thereof can be applied in different / corresponding audio contexts, thus generating a large number of combinations forming audio context-dependent control functions.

[0237] In addition to the VoIP and non-VoIP contexts distinguished and utilized in formulas (17) and (18), other audio contexts such as cinematic media, long-term music and games or low quality audio and high quality audio can be utilized in a similar manner.

[0238] (Section 4.3 Context Types) Context types can also be used directly to control a volume leveler to avoid annoying sounds like noise being boosted too much. For example, a VoIP confidence value can be used to steer a volume leveler to be less sensitive when the confidence value is high.

[0239] Specifically, the VoIP trust value Conf VOIPThe volume leveller level is (1-Conf VOIP ), i.e. the volume leveler is mostly disabled for VoIP content (when the VoIP confidence value is high). This is consistent with the traditional manual setup (preset) of disabling the volume leveler for VoIP contexts.

[0240] Alternatively, different dynamic gain ranges can be set for different contexts of the audio signal. In general, the VL (volume leveller) amount further adjusts the amount of gain applied to the audio signal and can be seen as another (non-linear) weight on the gain. In one embodiment, the setup can be as follows:

[0241] [Table 1] Further, assume that the expected VL amount is predefined for each context type. For example, the VL amount is set as 1 for cinematic media, 0 for VoIP, 0.6 for music, and 0.3 for gaming, but the present application is not limited thereto. According to this example, if the dynamic gain range for cinematic media is 100%, the dynamic gain range for VoIP is 60%, and so on. If the classification of the audio classifier 200 is based on hard decisions, the dynamic gain range may be directly set as in the above example. If the classification of the audio classifier 200 is based on soft decisions, the range may be adjusted based on the confidence value of the context type.

[0242] Similarly, the audio classifier 200 may identify multiple context types from the audio signal, and the adjustment unit 300C may be configured to adjust the dynamic gain range by weighting the confidence values ​​of the multiple content types based on the importance of the multiple content types.

[0243] In general, for context types, functions similar to (10)-(16) can be used here, with the content types replaced by the context types, to adaptively set the appropriate VL amounts. Indeed, Table 1 reflects the importance of different context types.

[0244] From another perspective, the confidence values ​​may be used to derive the normalized weights discussed in Section 1.4. Given that certain quantities are predefined for each context type in Table 1, a formula similar to formula (9) may also be applied. Note that a similar solution may be applied to multiple content types and any other audio types.

[0245] <Section 4.4 Combinations of embodiments and application scenarios> As in Part 1, all of the embodiments and variations thereof discussed above may be implemented in any combination thereof, and any components mentioned in different parts / embodiments but having the same or similar functions may be implemented as the same or separate components. For example, any two or more of the solutions described in Sections 4.1 to 4.3 may be combined with each other. And any of these combinations may be further combined with any of the embodiments described or implied in Parts 1 to 3 and other parts described below.

[0246] FIG. 21 shows the effect of the volume leveler controller proposed in this application by comparing the original short-term segment (FIG. 21(A)), the short-term segment processed by a conventional volume leveler without parameter modification (FIG. 21(B)), and the short-term segment processed by the volume leveler presented in this application (FIG. 21(C)). As can be seen, with the conventional volume leveler shown in FIG. 21(B), the volume of the noise (later half of the audio signal) is also boosted, which is annoying. In contrast, with the new volume leveler shown in FIG. 21(C), the volume of the useful part of the audio signal is boosted without apparently boosting the volume of the noise, which gives a good experience to the listener.

[0247] <Section 4.5 Volume Leveler Control Method> As in the first part, in the course of describing the volume leveller controller in the above embodiment, it is clear that several processes or methods are also disclosed. In the following, an overview of these methods is given without repeating some of the details already discussed above.

[0248] First, the embodiment of the audio processing method discussed in Part 1 may be used for a volume leveller, where the parameter(s) of the volume leveller is one of the targets to be adjusted by the audio processing method. In this respect, the audio processing method is also a volume leveller control method.

[0249] In this section only the specific aspects of the control of the volume leveller are discussed, for general aspects of the control method reference can be made to part 1.

[0250] According to the present application, a volume leveller control method is provided that identifies a content type of an audio signal in real time and adjusts the volume leveller in a continuous manner based on the identified content type by positively correlating a dynamic gain of the volume leveller with an informational content type of the audio signal and negatively correlating the dynamic gain of the volume leveller with an interfering content type of the audio signal.

[0251] Content types may include speech, short-term music, noise and background sounds. Generally, noise is considered an interfering content type.

[0252] When adjusting the dynamic gain of the volume leveler, it may be adjusted directly based on the confidence value of the content type, or it may be adjusted via a transfer function of the confidence value of the content type.

[0253] As already mentioned, an audio signal may be classified into multiple audio types simultaneously. When multiple content types are concerned, the adjustment operation 1104 may be configured to take into account at least some of the multiple audio content types through weighting the confidence values ​​of the multiple content types based on the importance of the multiple content types or through weighting the effect of the multiple content types based on the confidence values. In particular, the adjustment operation 1104 may be configured to take into account at least one dominant content type based on the confidence values. When the audio signal includes both interfering content type(s) and informational content type(s), the adjustment operation may be configured to take into account at least one dominant interfering content type based on the confidence values ​​and / or to take into account at least one dominant informational content type based on the confidence values.

[0254] Different audio types may affect each other, therefore the adjusting operation 1104 may be configured to modify the weight of one content type using the confidence value of at least one other content type.

[0255] The confidence values ​​of the audio type of the audio signal may be smoothed as described in part 1. For details of the smoothing operation, please refer to part 1.

[0256] The method may further include identifying a context type of the audio signal, where the adjusting operation 1104 may be configured to adjust a range of the dynamic gain based on the confidence value of the context type.

[0257] The role of a content type is restricted by the context type in which it is located. Thus, when both content type information and context type information are obtained simultaneously for an audio signal (i.e., for the same audio segment), the content type of the audio signal may be determined to be informative or interfering based on the context type of the audio signal. Furthermore, content types in an audio signal of different context types may be assigned different weights depending on the context type of the audio signal. From another perspective, different weights (larger or smaller, positive or negative values) can be used to reflect the informative or interfering nature of the content type.

[0258] Context types of audio signals may include VoIP, cinematic media, long-term music, and games. In a contextual VoIP audio signal, background sound may be considered an interfering content type, while in a contextual non-VoIP audio signal, background and / or speech and / or music; are considered informational content types. Other context types may include high quality or low quality audio.

[0259] When an audio signal is classified into multiple context type information simultaneously (i.e. for the same audio segment) as multiple content types, the adjustment operation 1104 may be configured to take into account at least some of the multiple audio content types through weighting the confidence values ​​of the multiple context types based on the importance of the multiple context types or through weighting the effect of the multiple context types based on the confidence values. In particular, the adjustment operation may be configured to take into account at least one dominant content type based on the confidence values.

[0260] Finally, the method embodiments discussed in this section may use audio classification methods discussed in Parts 6 and 7, which are not described in detail here.

[0261] As with the audio processing device embodiments, any combination of the audio processing method embodiments and variations thereof is feasible. On the other hand, any aspect of the audio processing method embodiments and variations thereof may be separate solutions. Furthermore, any two or more solutions described in this section may be combined with each other, and these combinations may further be combined with any embodiment described or implied in other parts of this disclosure.

[0262] Part 5: Equalizer controller and control method Equalization is typically applied to a musical signal to adjust or modify its spectral balance, known as "tone" or "timbre." Traditional equalizers allow the user to configure the overall profile (curve or shape) of the frequency response (gain) in individual frequency bands to emphasize certain instruments or eliminate undesired sounds. Popular music players such as Windows Media Player provide graphic equalizers to adjust the gain in each frequency band to get the best listening experience for different genres of music, and also provide a collection of equalizer presets for different music genres such as rock, rap, jazz, and folk. Once a preset is selected and a profile is set, the same equalization gain is applied to the signal until the profile is manually modified.

[0263] In contrast, a dynamic equalizer provides a means to automatically adjust the equalization gain in each frequency band to preserve an overall consistency of the spectral balance with respect to a desired timbre or tone. This consistency is achieved by continuously monitoring the spectral balance of the audio, comparing it to a desired preset spectral balance, and dynamically adjusting the applied equalization gain to transform the original spectral balance of the audio to the desired spectral balance. The desired spectral balance is either manually selected or preset prior to processing.

[0264] Both kinds of equalizers share the following disadvantages: the best equalization profile, the desired spectral balance or related parameters need to be selected manually and cannot be automatically modified based on the audio content during playback. Determining the audio content type is very important to provide a good overall quality for various audio signals. Different music pieces, for example those of different genres, require different equalization profiles.

[0265] In an equalizer system where any kind of audio signal can be input (not just music), the equalizer parameters need to be adjusted based on the content type. For example, an equalizer is typically enabled for music signals, but disabled for speech signals, as it can change the timbre of the speech too much, making the signal sound unnatural.

[0266] To address this problem at least partially, the present application proposes to control the equalizer based on the embodiments discussed in the first part.

[0267] As with parts 2-4, as a specific example of the audio processing device and method discussed in part 1, the equalizer 408 may use all the embodiments discussed in part 1 and any combination of those embodiments disclosed therein. In particular, when controlling the equalizer 408, the audio classifier 200 and the adjustment unit 300 in the audio processing device 100 as shown in Figures 1-10 may constitute an equalizer 408 controller 2000 as shown in Figure 22. In this embodiment, the adjustment unit is specific to the equalizer 408 and may be referred to as 300D.

[0268] That is, based on the disclosure in the first part, the equalizer controller 2200 may have an audio classifier 200 for continuously identifying an audio type of an audio signal, and an adjustment unit 300D for adjusting the equalizer in a continuous manner based on a confidence value of the identified audio type. Similarly, the audio classifier 200 may include at least one of an audio content classifier 202 and an audio context classifier 204, and the volume equalizer controller 2200 may further include at least one of a type smoothing unit 712, a parameter smoothing unit 814, and a timer 916.

[0269] Therefore, in this section we will not repeat what has already been described in the first section, but will simply provide some specific examples of this section.

[0270] <Section 5.1 Content-type-based control> In general, for common audio content types such as music, speech, background sounds, and noise, the equalizer should be set differently for different content types. As in a traditional setup, the equalizer can be automatically enabled for music signals but disabled for speech. Or in a more continuous manner, a high equalization level can be set for music signals and a low equalization level for speech signals. In this way, the equalization levels of the equalizer can be automatically set for different audio contents.

[0271] It is observed that, especially for music, the equalizer does not work very well for pieces of music with dominant sources, since if inappropriate equalization is applied, the timbre of the dominant source may change significantly and sound unnatural. Considering this, it is better to set a low playing level for pieces of music with dominant sources, while the equalization level can be kept high for pieces of music without dominant sources. With this information, the equalizer can automatically set the equalization level for different music contents.

[0272] Music can also be grouped based on various attributes such as genre, instruments and general musical characteristics including rhythm, tempo and timbre. Just as different equalizer presets are used for different music genres, these music groups / clusters may also have their own optimal equalization profile or equalizer curve (in the case of traditional equalizers) or optimal desired spectral balance (in the case of dynamic equalizers).

[0273] As mentioned above, the equalizer is typically enabled for music content but disabled for speech, since the equalizer may make the dialogue sound less good due to timbre changes. One way to achieve this automatically is to relate the equalizer level to the content, in particular to the music confidence value and / or the speech confidence value obtained from the audio content classification module. Here, the equalization level can be described as the weight of the equalizer gain applied. The higher the level, the stronger the equalization applied. For example, an equalization level of 1 applies a full equalization profile. An equalization level of 0 corresponds to all gains being 0 dB, and thus no equalization is applied. The equalization level may be represented by different parameters in different implementations of the equalizer algorithm. An example of this parameter is the equalizer weight as implemented in US Pat. No. 6,399,433, which is incorporated herein by reference in its entirety.

[0274] Various control methods can be designed to adjust the equalization level. For example, in the audio content type information, the speech confidence value or the music confidence value is used to set the equalization level. L eq ∝Conf music (20) or L eq ∝1-Conf speech (twenty one) where L eq is the equalization level, and Conf music and Conf speechrepresents the confidence value of music and speech.

[0275] That is, adjustment unit 300D may be configured to positively correlate the equalization level with the short-term music confidence value or negatively correlate the equalization level with the speech confidence value.

[0276] The speech confidence value and the music confidence value can further be jointly utilized to set the equalization level. The general idea is that the equalization level is high only when the music confidence value is high and the speech confidence value is low, otherwise the equalization level is low. For example, L eq =Conf music (1-Conf speech α ) (twenty two) Here, the speech confidence value is raised to the power of α to handle frequently occurring non-zero speech confidence values ​​in music signals. Using the above formula, equalization is fully applied (at a level equal to 1) for pure music signals without speech components. As mentioned in Part 1, α may be considered as a weighting factor based on the importance of the content type and can typically be set to 1 or 2.

[0277] If more weight is placed on the confidence value of an utterance, the adjustment unit 300D may be configured to disable the equalizer 408 when the confidence value for the content-type utterance is above a certain threshold.

[0278] In the above description, music and speech content types are taken as examples. Alternatively or additionally, background sound and / or noise confidence values ​​may also be considered. In particular, the adjustment unit 300D may be configured to positively correlate the equalization level with the background confidence value and / or negatively correlate the equalization level with the noise confidence value.

[0279] As another example, the confidence values ​​may be used to derive the normalized weights discussed in Section 1.4. Given that the expected equalization levels are predefined for each content type (e.g., 1 for music, 0 for speech, 0.5 for noise and background), a formula similar to formula (8) can be applied exactly.

[0280] The equalization levels may further be smoothed to avoid abrupt changes that may introduce audible artifacts at the transition points. This can be done using the parameter smoothing unit 814 described in section 1.5.

[0281] Section 5.2: Predominant source certainty in music To avoid music with dominant sources being subjected to a high equalization level, the equalization level is further determined by a confidence value Conf, which indicates whether the music piece contains a dominant source. dom For example, L eq =1-Conf dom (twenty three).

[0282] In this way, the equalization level is low for music pieces with dominant sources and high for music pieces without dominant sources.

[0283] Although confidence values ​​for music with dominant sources are described here, confidence values ​​for music without dominant sources can also be used, i.e., the adjustment unit 300D may be configured to make the equalization level positively correlated with the confidence values ​​for short-term music without dominant sources and / or to make the equalization level negatively correlated with the confidence values ​​for short-term music with dominant sources.

[0284] As mentioned in Section 1.1, music and speech, and music with or without dominant sources, are content types at different hierarchical levels, but can be considered in parallel. By jointly considering the confidence values ​​of the dominant source and the confidence values ​​of speech and music as above, the equalization level can be set by combining at least one of formulas (20)-(21) with (23). One example is combining all three formulas as L eq =Conf music (1-Conf speech )(1-Conf dom ) (twenty four) The idea is to do so.

[0285] For generality, different weights based on the importance of the content type can further be applied to the different confidence values, as in formula (22).

[0286] As another example, Conf dom Assuming that x is calculated only when the audio signal is music, the step function can be designed as follows:

[0287]

number

[0288] The same smoothing scheme discussed in section 1.5 can also be applied, and the time constant α can further be set based on the transition type, such as a transition from music with a dominant source to music without a dominant source or from music without a dominant source to music with a dominant source. For this purpose, a formula similar to formula (4') can also be applied.

[0289] (Section 5.3 Equalizer Preset) In addition to adaptively adjusting the equalization level based on the confidence value of the audio content type, appropriate equalization profiles or desired spectral balance presets can also be automatically selected for different audio content depending on its genre, instrument, or other characteristics. Music of the same genre, containing the same instrument, or having the same musical characteristics can share the same equalization profile or desired spectral balance preset.

[0290] For generality, we use the term "music cluster" to represent a group of music with the same genre, the same instrument, or similar musical attributes. This can be considered as another hierarchical level of the audio content types described in section 1.1. An appropriate equalization profile, equalization level, and / or desired spectral balance preset may be associated with each music cluster. An equalization profile is a gain curve applied to the music signal and can be any of the equalizer presets used for different music genres (classical, rock, jazz, and folk, etc.), and a desired spectral balance preset represents the desired timbre for each cluster. Figure 23 shows some examples of desired spectral balance presets implemented in Dolby Home Theater technology. Each describes a desired spectral shape across the audible frequency range. This shape is successively compared to the spectral shape of the incoming audio, and an equalization gain is calculated from the comparison of the transformation of the incoming audio's spectral shape to the preset spectral shape.

[0291] For a new piece of music, the closest cluster can be determined (hard decision) or a confidence value for each music cluster can be calculated (soft decision). Based on this information, an appropriate equalization profile or a desired spectral balance preset can be determined for a given piece of music. The simplest way is to use P eq =P c* (26) The goal is to assign the corresponding profile of the best matching cluster as eq is the estimated equalization profile or desired spectral balance preset, and c * is the index of the best matching music cluster (dominant audio type), which is obtained by picking the cluster with the highest confidence value.

[0292] Furthermore, there may be more than one music cluster with a confidence value greater than 0, i.e. a music piece has more or less similar attributes to its clusters. For example, a music piece may have multiple instruments, or multiple genre attributes. This inspires another way to estimate the appropriate equalization profile by considering all clusters, instead of using only the closest cluster. For example, a weighted sum can be used.

[0293]

number

[0294] For some applications, it may not be desirable to involve all the clusters given in formula (27), since only a subset of the clusters needs to be considered - those most relevant to the current piece of music - and formula (27) can be slightly modified as follows:

[0295]

number

[0296] In the above description, music clusters are taken as an example. In fact, these solutions are applicable to audio types at any hierarchical level discussed in section 1.1. Thus, in general, the adjustment unit 300D may be configured to assign an equalization level and / or an equalization profile and / or a spectral balance preset to each audio type.

[0297] <Section 5.4 Context-based control> The previous sections have focused on various content types. In further embodiments discussed in this section, context types may alternatively or additionally be considered.

[0298] Typically, the equalizer is enabled for music but disabled for cinematic media, since the obvious timbre changes can make dialogue in cinematic media sound less good. This shows that the equalization level can be related to the long-term music confidence value and / or the cinematic media confidence value: L eq ∝Conf MUSIC (29) or L eq ∝1-Conf MOVIE (30) Here, L eq is the equalization level, Conf MUSIC and Conf MOVIE represents the long-term music and movie media trust value.

[0299] That is, adjustment unit 300D may be configured to make the equalization level positively correlated with the long-term music confidence value or the equalization level negatively correlated with the cinematic media confidence value.

[0300] That is, for a cinematic media signal, the cinematic media confidence value is high (or the music confidence is low) and thus the equalization level is low, whereas for a music signal, the cinematic media confidence value is low (or the music confidence is high) and thus the equalization level is high.

[0301] The solutions shown in formulas (29) and (30) may be modified in a similar manner to formulas (22) through (25) and / or may be combined with any of the solutions shown in formulas (22) through (25).

[0302] Additionally or alternatively, the tuning unit 300D may be configured to negatively correlate the equalization level with the confidence value of the game.

[0303] In another embodiment, the confidence values ​​may be used to derive the normalized weights discussed in Section 1.4. Given that the expected equalization level / profile is predefined for each context type (the equalization profiles are shown in Table 2 below), a formula similar to formula (9) may also be applied.

[0304] [Table 2] Here, in some profiles, all gains can be set to 0 as a way of disabling the equalizer for certain context types such as cinematic media and games.

[0305] <Section 5.5 Combinations of embodiments and application scenarios> As with Part 1, all of the embodiments and variations thereof discussed above may be implemented in any combination thereof, and any components mentioned in different parts / embodiments but having the same or similar functionality may be implemented as the same or separate components.

[0306] For example, any two or more of the solutions described in Sections 5.1 to 5.4 may be combined with each other, and any of these combinations may be further combined with any of the embodiments described or implied in Parts 1 to 4 and other parts described below.

[0307] <Section 5.6 Equalizer control method> As in the first part, in the course of describing the equalizer controller in the above embodiment, it is clear that several processes or methods are also disclosed. In the following, an overview of these methods is given without repeating some of the details already discussed above.

[0308] First, the embodiment of the audio processing method discussed in the first part may be used for an equalizer, whose parameter(s) are one of the targets to be adjusted by the audio processing method. From this point of view, the audio processing method is also an equalizer control method.

[0309] In this section only the specific aspects of the control of the equalizer are discussed, for general aspects of the control method reference can be made to Part 1.

[0310] According to embodiments, an equalizer control method may include identifying an audio type of an audio signal in real time and adjusting an equalizer in a continuous manner based on a confidence value of the identified audio type.

[0311] As elsewhere in this application, when multiple audio types with corresponding confidence values ​​are involved, the adjusting operation 1104 may be configured to take into account at least some of the multiple audio types through weighting the confidence values ​​of the multiple audio types based on the importance of the multiple audio types or through weighting the effect of the multiple audio types based on the confidence values. In particular, the adjusting operation 1104 may be configured to take into account at least one dominant content type based on the confidence values.

[0312] The adjusted parameter values ​​may be smoothed as described in Part 1. See Sections 1.5 and 1.8, which will not be described in detail here.

[0313] The audio type may be content type or context type. When a content type is concerned, the adjustment operation 1104 may be configured to positively correlate the equalization level with the short-term music confidence value and / or negatively correlate the equalization level with the speech confidence value. Additionally or alternatively, the adjustment operation may be configured to positively correlate the equalization level with the background confidence value and / or negatively correlate the equalization level with the noise confidence value.

[0314] When concerned with a context type, the adjustment operation 1104 may be configured to positively correlate the equalization level with a long-term music confidence value and / or negatively correlate the equalization level with a cinematic media and / or gaming confidence value.

[0315] For short-term music content types, the adjustment operation 1104 may be configured to positively correlate the equalization level with the confidence value of short-term music without a dominant source and / or negatively correlate the equalization level with the confidence value of short-term music with a dominant source, which can only be done when the confidence value for short-term music is greater than a certain threshold.

[0316] In addition to adjusting the equalization level, other aspects of the equalizer may be adjusted based on the confidence value(s) of the audio type(s) of the audio signal. For example, the adjusting operation 1104 may be configured to assign an equalization level and / or an equalization profile and / or a spectral balance preset to each audio type.

[0317] For specific examples of audio types, see Part 1.

[0318] As with the audio processing device embodiments, any combination of the audio processing method embodiments and variations thereof is feasible. On the other hand, any aspect of the audio processing method embodiments and variations thereof may be separate solutions. Furthermore, any two or more solutions described in this section may be combined with each other, and these combinations may further be combined with any embodiment described or implied in other parts of this disclosure.

[0319] Part VI: Audio classifiers and classification methods As mentioned in Sections 1.1 and 1.2, the audio types discussed in this application, including content types and context types at various hierarchical levels, can be classified or identified using any existing classification scheme, including machine learning based methods. In this and the next part, this application proposes some novel aspects of classifiers and methods for classifying the context types mentioned in the previous parts.

[0320] <Section 6.1 Context Classifiers Based on Content Type Classification> As mentioned in the previous sections, the audio classifier 200 is used to identify a content type of an audio signal and / or to identify a context type of an audio signal. Thus, the audio classifier 200 may comprise an audio content classifier 202 and / or an audio context classifier 204. When adopting existing techniques for implementing the audio content classifier 202 and / or the audio context classifier 204, both classifiers may be independent of each other but may share some features and therefore some methods for extracting said features.

[0321] In this part and in the following seventh part, according to the novel aspects proposed in the present application, the audio context classifier 204 may utilize the result of the audio content classifier 202. That is, the audio classifier 200 comprises an audio content classifier 202 for identifying a content type of the audio signal; and an audio context classifier 204 for identifying a context type of the audio signal based on the result of the audio content classifier 202. Thus, the classification result of the audio content classifier 202 may be used by both the audio context classifier 204 and the adjustment unit 300 (or adjustment units 300A to 300D) discussed in the previous parts. However, although not shown in the drawings, the audio classifier 200 may include two audio content classifiers 202 used by the adjustment unit 300 and the audio context classifier 204, respectively.

[0322] Furthermore, as discussed in Section 1.2, especially when classifying multiple audio types, the audio content classifier 202 or the audio context classifier 204 may be a group of classifiers that work together, although they can also be implemented as one single classifier.

[0323] As discussed in Section 1.1, content types are audio type categories for short-term audio segments, typically with lengths on the order of a few frames or tens of frames (e.g., 1 second), and context types are audio type categories for long-term audio segments, typically with lengths on the order of a few seconds or tens of seconds (e.g., 10 seconds). Thus, we use "short-term" and "long-term" for "content type" and "context type", respectively, when necessary. However, as discussed in the next Part 7, context types, although they indicate the attributes of the audio signal on a relatively long time scale, can also be identified based on features extracted from short-term audio segments.

[0324] Turning now to the structure of the audio content classifier 202 and the audio context classifier 204 with reference to FIG.

[0325] As shown in Fig. 24, the audio content classifier 202 may include a short-term feature extractor 2022 for extracting short-term features from short-term audio segments, each of which includes a sequence of audio frames; and a short-term classifier 2024 for classifying a sequence of short-term segments in a long-term audio segment into short-term audio types using their respective short-term features. Both the short-term feature extractor 2022 and the short-term classifier 2024 may be implemented using existing techniques, although some modifications are also proposed for the short-term feature extractor 2022 in section 6.3 below.

[0326] The short-term classifier 2024 may be configured to classify each segment of the sequence of short-term segments into at least one of the following short-term audio types (content types): speech, short-term music, background sounds, and noise, as described in Section 1.1. Each content type may be further classified into content types at lower hierarchical levels, such as, but not limited to, those discussed in Section 1.1.

[0327] As known in the art, confidence values ​​for the classified audio types may be obtained by a short-term classifier 2024. When referring to the operation of any classifier in this application, whether explicitly recorded or not, it is understood that a confidence value is obtained contemporaneously if necessary. Examples of audio type classification may be found in "Audio Type Classification: A Short-Term Classifier 2024," which is incorporated herein by reference in its entirety.

[0328] On the other hand, as shown in Fig. 24, the audio context classifier 204 may comprise a statistics extractor 2024 for computing statistics of the results of the short-term classifier for a sequence of short-term segments in a long-term audio segment as long-term features; and a long-term classifier 2044 for classifying the long-term audio segment into a long-term audio type using the long-term features. Similarly, both the statistics extractor 2042 and the long-term classifier 2044 may be implemented using existing techniques, although some modifications are proposed for the statistics extractor 2042 in the following section 6.2.

[0329] The long-term classifier 2044 may be configured to classify the long-term audio segments into at least one of the following long-term audio types (context types): cinematic media, long-term music, gaming, and VoIP, as described in section 1.1. Alternatively or additionally, the long-term classifier 2044 may be configured to classify the long-term audio segments into VoIP or non-VoIP, as described in section 1.1. Alternatively or additionally, the long-term classifier 2044 may be configured to classify the long-term audio segments into high quality audio or low quality audio, as described in section 1.1. In practice, different target audio types can be selected and trained based on the needs of the application / system.

[0330] Regarding the meaning and selection of short-term and long-term segments (as well as frames as discussed in Section 6.3), reference may be made to Section 1.1.

[0331] <Section 6.2 Extracting Long-Term Features> 24, in one embodiment only the statistics extractor 2042 is used to extract long-term features from the results of the short-term classifier 2024. As long-term features at least one of the following may be calculated by the statistics extractor 2042: the mean and variance of the confidence values ​​of the short-term audio types of the short-term segments within the long-term segment to be classified, said mean and variance weighted by the importance of the short-term segment, the frequency of occurrence of each short-term audio type and the frequency of transitions between different short-term audio types within the long-term segment to be classified.

[0332] In Fig. 25 we show the average speech and short-term music confidence values ​​for each short-term segment (length 1 s). For comparison, segments have been extracted from three different audio contexts: cinematic media (Fig. 25(A)), long-term music (Fig. 25(B)) and VoIP (Fig. 25(C)). For the cinematic media context, it can be observed that high confidence values ​​are obtained for either the speech type or the music type, with frequent alternations between these two audio types. In contrast, the long-term music segments give stable high short-term music confidence values ​​and relatively stable low speech confidence values. On the other hand, the VoIP segments give stable low short-term music confidence values, but fluctuating speech confidence values ​​due to pauses during the VoIP conversation.

[0333] The variance of the confidence values ​​for each audio type is also an important feature for classifying different audio contexts. Figure 26 gives histograms of the variance of the confidence values ​​of speech, short-term music, background and noise in cinematic media, long-term music and VoIP audio contexts (the horizontal axis is the variance of the confidence values ​​in the dataset, and the vertical axis is the number of occurrences in each bin of the variance value s in the dataset, which can be normalized to show the normal probability of each bin of the variance value). For cinematic media, the variances of the confidence values ​​of speech, short-term music and background are all relatively high and widely distributed, indicating that the confidence values ​​of these audio types are changing strongly. For long-term music, the variances of the confidence values ​​of speech, short-term music, background and noise are all relatively low and narrowly distributed, indicating that the confidence values ​​of these audio types remain stable. The speech confidence values ​​remain consistently low and the music confidence values ​​remain consistently high. For VoIP, the variance of the confidence values ​​of short-term music is low and narrowly distributed, while the variance of the confidence values ​​of speech is relatively widely distributed. This is due to the frequent pauses during a VoIP conversation.

[0334] The weights used in calculating the weighted mean and variance are determined based on the importance of each short-term segment, which may be measured by its energy or loudness, which can be estimated using many existing techniques.

[0335] The frequency of occurrence of each short-term audio type in the long-term segment to be classified is the count of each audio type within the long-term segment to which the short-term segments were classified, normalized by the length of the long-term segment.

[0336] The frequency of transitions between different short-term audio types within a long-term segment to be classified is the count of audio type changes between adjacent short-term segments within the long-term segment to be classified, normalized by the length of the long-term segment.

[0337] When discussing the mean and variance of the confidence values ​​with reference to Fig. 25, the frequency of occurrence of each short-term audio type and the transition frequency between those various short-term audio types are actually touched upon. These features are also highly relevant for audio context classification. For example, long-term music contains mostly short-term music audio types, and thus has a high frequency of short-term music. On the other hand, VoIP contains mostly speech and pauses, and thus has a high frequency of speech or noise. As another example, cinematic media transitions between different short-term audio types more frequently than long-term music or VoIP, and thus generally has a higher transition frequency between short-term music, speech, and background. VoIP typically transitions between speech and noise more frequently than others, and thus has a higher transition frequency between speech and noise.

[0338] In general, it is assumed that the long-term segments are of the same length in the same application / system. If so, the occurrence counts of each short-term audio type and the transition counts between different short-term audio types within a long-term segment may be used directly without normalization. If the length of the long-term segments is variable, the occurrence and transition frequencies mentioned above should be used. The claims of this application should be interpreted to cover both situations.

[0339] Additionally or alternatively, the audio classifier 200 (or the audio context classifier 204) may further include a long-term feature extractor 2046 (FIG. 27) for extracting further long-term features from the long-term audio segment based on the short-term features of the sequence of short-term segments in the long-term audio segment. In other words, the long-term feature extractor 2046 does not use the classification result of the short-term classifier 2024, but directly uses the short-term features extracted by the short-term feature extractor 2022 to derive some long-term features to be used by the long-term classifier 2044. The long-term feature extractor 2046 and the statistics extractor 2042 may be used independently or jointly. In other words, the audio classifier 200 may include one or both of the long-term feature extractor 2046 or the statistics extractor 2042.

[0340] Any feature can be extracted by the long-term feature extractor 2046. In this application, it is proposed to calculate as long-term features at least one of the following statistics of the short-term features from the short-term feature extractor 2022: mean, variance, weighted mean, weighted variance, high mean, low mean and the ratio (contrast) between high mean and low mean.

[0341] The mean and variance of the short-term features extracted from the short-term segments within the long-term segment to be classified.

[0342] The weighted mean and variance of the short-term features extracted from the short-term segments within the long-term segment to be classified. The short-term features are weighted based on the importance of each short-term segment, measured using its energy or loudness as just mentioned.

[0343] High average: The average of selected short-term features extracted from the short-term segments within the long-term segment to be classified. A short-term feature is selected when it meets at least one of the following conditions: greater than a certain threshold; not lower than all other short-term features; within a pre-determined percentage of short-term features, e.g., within the top 10% of short-term features.

[0344] Low average: The average of selected short-term features extracted from the short-term segments within the long-term segment to be classified. A short-term feature is selected when it meets at least one of the following conditions: less than a certain threshold; not higher than all other short-term features; within a pre-determined percentage of the short-term features, e.g., within the bottom 10% of the short-term features.

[0345] Contrast: The ratio between the high and low averages. It represents the dynamics of short-term features within a long-term segment.

[0346] The short-term feature extractor 2022 may be implemented using existing techniques, by which any features can be extracted. Nevertheless, some modifications for the short-term feature extractor 2022 are proposed in the following section 6.3.

[0347] Section 6.3 Extracting Short-Term Features As shown in Figures 24 and 27, the short-term feature extractor 2022 may be configured to extract at least one of the following features as short-term features directly from each short-term audio segment: rhythmic features, pause / mute features and short-term audio quality features.

[0348] Rhythmic characteristics may include rhythmic strength, rhythmic regularity, rhythmic clarity (see non-patent document 2, which is incorporated by reference herein in its entirety), and 2D subband modulation (see non-patent document 3, which is incorporated by reference herein in its entirety).

[0349] The pause / mute characteristics may include speech pause, sharp decay, mute length, unnatural silence, average unnatural silence, and total energy of unnatural silence.

[0350] Short-term audio quality features are audio quality features related to short-term segments, which are similar to the audio quality features extracted from audio frames, discussed below.

[0351] Alternatively or additionally, as shown in Fig. 28, the audio classifier 200 may comprise a frame level feature extractor 2012 for extracting frame level features from each frame of a sequence of audio frames comprised in a short-term segment. A short-term feature extractor 2022 may be configured to compute short-term features based on the frame level features extracted from the sequence of audio frames.

[0352] As a pre-processing, the input audio signal may be down-mixed to a mono audio signal. This pre-processing is not necessary if the audio signal is already a mono signal. It is then divided into frames with a predefined length (typically 10 to 25 ms). Correspondingly, frame-level features are extracted from each frame.

[0353] The frame-level feature extractor 2012 may be configured to extract at least one of the following features: features characterizing various short-term audio type attributes, cut-off frequencies, static signal-to-noise ratio (SNR) characteristics, segmental signal-to-noise ratio (SNR) characteristics, fundamental speech descriptors, and vocal tract characteristics.

[0354] The features characterizing the attributes of various short-term audio types (particularly speech, short-term music, background sounds and noise) may include at least one of the following features: frame energy, subband spectral distribution, spectral flux, Mel-frequency Cepstral Coefficients (MFCCs), bass, residual information, Chroma features and zero-crossing rate.

[0355] For details on MFCCs, see "Color-enhanced 3D Chroma Recognition," in ...

[0356] The cutoff frequency represents the highest frequency in an audio signal above which the content energy is close to zero. It is designed to detect band-limited content, which is useful here for audio context classification. The cutoff frequency is usually caused by the coding, since most coders discard high frequencies at low or medium bitrates. For example, the MP3 codec has a cutoff frequency of 16 kHz at 128 kbps. As another example, many common VoIP codecs have cutoff frequencies of 8 kHz or 16 kHz.

[0357] Besides the cutoff frequency, signal degradation during the audio encoding process is considered as an additional characteristic to distinguish various audio contexts, such as VoIP vs. non-VoIP contexts, high quality vs. low quality audio contexts. To capture richer characteristics, features representative of audio quality, such as those for objective speech quality assessment (see non-patent document 5, which is incorporated herein by reference in its entirety), may be further extracted at multiple levels. Examples of audio quality features include: a) Static SNR characteristics including estimated background noise level, spectral clarity, etc. b) Segment SNR characteristics including spectral level deviation, spectral level range, relative noise floor, etc. c) Basic speech descriptors including pitch mean, speech section level variation, speech level, etc. d) Vocal tract characteristics including robotization, pitch cross power, etc.

[0358] To derive short-term features from the frame-level features, the short-term feature extractor 2022 may be configured to calculate statistics of the frame-level features as the short-term features.

[0359] Examples of frame-level feature statistics include mean and standard deviation, which capture rhythmic attributes to distinguish between various audio types such as short-term music, speech, background, and noise. For example, speech typically alternates between voiced and unvoiced sounds at a syllabic rate, while music does not have such alternations, and the variance of frame-level features for speech is typically larger than for music.

[0360] Another example of a statistic is a weighted average of frame-level features, e.g., a cutoff frequency. The weighted average of the cutoff frequencies derived from all audio frames in a short-term segment, weighted by the energy or loudness of each frame, gives the cutoff frequency for the short-term segment.

[0361] Alternatively or additionally, as shown in Fig. 29, the audio classifier 200 may comprise a frame level feature extractor 2012 for extracting frame level features from audio frames and a frame level classifier 2014 for classifying each frame of a sequence of audio frames into a frame level audio type using the respective frame level features, wherein the short-term feature extractor 2022 may be configured to compute short-term features based on the results of the frame level classifier 2014 for the audio frames of the sequence.

[0362] In other words, in addition to the audio content classifier 202 and the audio context classifier 204, the audio classifier 200 may further include a frame classifier 201. In such a configuration, the audio content classifier 202 classifies short-term segments based on the frame-level classification result of the frame classifier 201, and the audio context classifier 204 classifies long-term segments based on the short-term classification result of the audio content classifier 202.

[0363] The frame-level classifier 2014 may be configured to classify each frame of a sequence of audio frames into a class, which may be referred to as a "frame-level audio type." In an embodiment, the frame-level audio type may have a similar configuration and may have a similar meaning as the content type configuration discussed above. The only difference is that the frame-level audio type and the content type are classified at different levels of the audio signal, i.e., at the frame level and the short-term segment level. For example, the frame-level classifier 2014 may be configured to classify each frame of a sequence of audio frames into at least one of the following frame-level audio types: speech, music, background sounds, and noise. On the other hand, the frame-level audio type may have a configuration that is partially or completely different from the content type configuration, which is more suitable for frame-level classification and is more suitable to be used as a short-term feature for short-term classification. For example, the frame-level classifier 2014 may be configured to classify each frame of a sequence of audio frames into at least one of the following frame-level audio types: voiced, unvoiced, and pause.

[0364] A similar approach may be adopted by referring to the description in Section 6.2 on how to derive short-term features from the results of frame-level classification.

[0365] Alternatively, short-term features based on the results of the frame level classifier 2014 and short-term features directly based on the frame level features obtained by the frame level feature extractor 2012 may be used by the short-term classifier 2024. Thus, the short-term feature extractor 2022 may be configured to compute short-term features based on both the frame level features extracted from the sequence of audio frames and the results of the frame level classifier on the sequence of audio frames.

[0366] In other words, the frame-level feature extractor 2012 may be configured to compute both statistics similar to those discussed in Section 6.2 and short-term features including at least one of the following features described in connection with FIG. 28: features characterizing various short-term audio type attributes, cutoff frequency, static signal-to-noise ratio characteristics, segmental signal-to-noise ratio characteristics, fundamental speech descriptors and vocal tract characteristics.

[0367] To work in real time, in all embodiments the short-term feature extractor 2022 may be configured to operate on the short-term audio segments formed using a moving window that slides in the time dimension of the long-term audio segment with a predefined step length, see section 1.1 for details on moving windows for short-term audio segments and moving windows for long-term audio segments.

[0368] <Section 6.4 Combinations of embodiments and application scenarios> As with Part 1, all of the embodiments and variations thereof discussed above may be implemented in any combination thereof, and any components mentioned in different parts / embodiments but having the same or similar functionality may be implemented as the same or separate components.

[0369] For example, any two or more of the solutions described in Sections 6.1 to 6.3 may be combined with each other, and any of these combinations may be further combined with any of the embodiments described or implied in Parts 1 to 5 and other parts described below. In particular, the type smoothing unit 712 discussed in Part 1 may be used in this part as a component of the audio classifier 200 to smooth the results of the frame classifier 2014 or the audio content classifier 202 or the audio context classifier 204. Furthermore, a timer 916 may also play a role as a component of the audio classifier 200 to avoid sudden changes in the output of the audio classifier 200.

[0370] <Section 6.5 Audio Classification Method> As in the first part, it is clear that in the course of describing the audio classifier in the above embodiment, several processes or methods are also disclosed. In the following, an overview of these methods is given without repeating some of the details already discussed above.

[0371] In one embodiment, shown in Fig. 30, a method for audio classification is provided. To identify the long-term audio type (i.e. content type) of a long-term audio segment consisting of a sequence of short-term audio segments (overlapping or non-overlapping), the short-term audio segments are first classified into short-term audio types, i.e. content types (operation 3004), and long-term features are obtained by calculating statistics of the results of the classification operation on the sequence of short-term segments in the long-term audio segment (operation 3006). A long-term classification (operation 3008) may then be performed using the short-term audio segments. The short-term audio segments may comprise a sequence of audio frames. Of course, in order to identify the short-term audio type of the short-term segments, short-term features need to be extracted from those segments.

[0372] Short-term audio types (content types) may include, but are not limited to, speech, short-term music, background sounds and noise.

[0373] The long-term features are the mean and variance of the confidence values ​​of those short-term segments, said mean and variance weighted by the importance of those short-term segments, the frequency of occurrence of each short-term audio type and the frequency of transitions between different short-term audio types.

[0374] At some mean, further long-term features may be derived (operation 3107) based directly on the short-term features of the sequence of short-term segments within the long-term audio segment, as shown in Figure 31. Such long-term features may include, but are not limited to, the following statistics of the short-term features: mean, variance, weighted mean, weighted variance, high mean, low mean and the ratio between the high mean and the low mean.

[0375] There are various methods to extract short-term features. One is to extract short-term features directly from the short-term audio segments to be classified. Such features include, but are not limited to, rhythmic features, pause / mute features, and short-term audio quality features.

[0376] The second method is to extract frame-level features from audio frames included in each short-term segment (operation 3201 in FIG. 32), and then calculate short-term features based on the frame-level features, for example, calculate statistics of the frame-level features as short-term features. The frame-level features may include, but are not limited to: features characterizing various short-term audio type attributes, cutoff frequency, static signal-to-noise ratio characteristics, segmental signal-to-noise ratio characteristics, basic speech descriptors, and vocal tract characteristics. The features characterizing various short-term audio type attributes may further include frame energy, subband spectral distribution, spectral flux, Mle-frequency Cepstral Coefficients (MFCCs), bass, residual information, Chroma features, and zero-crossing rate.

[0377] A third method is to extract short-term features in a manner similar to extracting long-term features: after extracting frame-level features from audio frames in the short-term segment to be classified (operation 3201), classify each audio frame into a frame-level audio type using the respective frame-level features (operation 32011 in FIG. 33); the short-term features may be extracted by computing the short-term features based on the frame-level audio types (optionally including confidence values) (operation 3002). The frame-level audio types may have similar attributes and configurations as the short-term audio types (content types), and may also include speech, music, background sounds, and noise.

[0378] The second and third methods above may be combined together as shown by the dashed arrow in FIG.

[0379] As discussed in Part 1, both the short-term and long-term audio segments may be sampled using a moving window, i.e. the operation of extracting short-term features (operation 3002) may be performed on the short-term audio segment formed using a moving window that slides in the time dimension of the long-term audio segment with a predetermined step length, and the operation of extracting long-term features (operation 3107) and the operation of computing short-term audio type statistics (operation 3006) may be performed on the long-term audio segment formed using a moving window that slides in the time dimension of the audio signal with a predetermined step length.

[0380] As with the audio processing device embodiments, any combination of the audio processing method embodiments and variations thereof is feasible. On the other hand, any aspect of the audio processing method embodiments and variations thereof may be separate solutions. Furthermore, any two or more solutions described in this section may be combined with each other, and these combinations may further be combined with any embodiments described or implied in other parts of this disclosure. In particular, as already discussed in section 6.4, audio-based smoothing and transition schemes may be part of the audio classification methods discussed here.

[0381] Part 7: VoIP classifiers and classification methods In part 6, a novel audio classifier is proposed for classifying audio signals into audio context types based at least in part on the results of the content type classification. In the embodiment discussed in part 6, long-term features are extracted from long-term segments of several seconds to tens of seconds in length. Thus, audio context classification may cause long latency. It is desired that audio contexts can be classified in real-time or near real-time, for example at a short-term segment level.

[0382] Section 7.1 Context Classification Based on Short-Term Segments Thus, as shown in Figure 34, an audio classifier 200A is provided having an audio content classifier 202A for identifying a content type of a short-term segment of an audio signal and an audio context classifier 204A for identifying a context type of the short-term segment based at least in part on the content type identified by the audio content classifier.

[0383] Here, the audio content classifier 202A may employ the techniques already described in Part 6, but may also employ different techniques discussed in Section 7.2 below. Also, the audio context classifier 204A may employ the techniques already described in Part 6, but with a difference that the context classifier 204A may directly use the results of the audio content classifier 202A, rather than using statistics of the results from the audio content classifier 202A, since both the audio context classifier 204 and the audio content classifier 202A classify the same short-term segments. Furthermore, similar to Part 6, in addition to the results from the audio content classifier 202A, the audio context classifier 204 may use other features directly extracted from the short-term segments. That is, the audio context classifier 204A may be configured to classify the short-term segments based on a machine learning model, using as features the confidence values ​​of the content types of the short-term segments and other features extracted from the short-term segments. For the features extracted from the short-term segments, reference may be made to Part 6.

[0384] The audio content classifier 200A may simultaneously label short-term segments as more audio types than VoIP speech / noise and / or non-VoIP speech / noise (VoIP speech / noise and / or non-VoIP speech / noise are discussed in Section 7.2 below), where each of the multiple audio types may have its own confidence value as discussed in Section 1.2. This can achieve better classification accuracy since richer information can be supplemented. For example, the joint information of the confidence values ​​of speech and short-term music reveals how likely the audio content is to be a mixture of speech and background music, and thus can be differentiated from pure VoIP content.

[0385] Section 7.2 Classification using VoIP speech and VoIP noise This aspect of the present application is particularly useful in VoIP / non-VoIP classification systems that are required to classify current short-term segments due to short decision latency.

[0386] For this purpose, an audio classifier 200A is specially designed for VoIP / non-VoIP classification, as shown in Fig. 34. To classify VoIP / non-VoIP, a VoIP speech classifier 2026 and / or a VoIP noise classifier are developed to generate intermediate results for the final robust VoIP / non-VoIP classification by the audio context classifier 204.

[0387] A VoIP short-term segment will alternately contain VoIP speech and VoIP noise. It is observed that high accuracy can be achieved for classifying short-term segments of speech as VoIP speech or non-VoIP speech, but not for classifying short-term segments of noise as VoIP noise or non-VoIP noise. Thus, it can be concluded that discrimination is blurred by directly classifying short-term segments as VoIP (including VoIP speech and VoIP noise, but VoIP speech and VoIP noise are not identified individually) and non-VoIP without considering the difference between speech and noise, and thus keeping the features of these two content types (speech and noise) mixed together.

[0388] It makes sense for a classifier to achieve a higher accuracy for VoIP speech / non-VoIP speech classification than for VoIP noise / non-VoIP noise classification because speech contains more information than noise and features such as cutoff frequency are more effective for classifying speech. According to the weight ranking obtained from the AdaBoost training / process, the top weighted short-term features for VoIP / non-VoIP classification are: standard deviation of logarithmic energy, cutoff frequency, standard deviation of rhythmic strength and standard deviation of spectral flux. The standard deviation of logarithmic energy, standard deviation of rhythmic strength and standard deviation of spectral flux are generally higher for VoIP speech than for non-VoIP speech. One likely reason is that many short-term speech segments in non-VoIP contexts such as cinematic media or games are usually mixed with other sounds such as background music or sound effects, for which the values ​​of the above features are lower. On the other hand, the cutoff feature will generally be lower for VoIP speech than for non-VoIP speech, which illustrates the low cutoff frequency introduced by many common VoIP codecs.

[0389] Thus, in one embodiment, the audio content classifier 202A may include a VoIP speech classifier 2026 for classifying the short-term segments as content-based VoIP speech or content-based non-VoIP speech, and the audio context classifier 204 may be configured to classify the short-term segments as contextual VoIP or contextual non-VoIP based on the confidence values ​​of the VoIP speech and non-VoIP speech.

[0390] In another embodiment, the audio content classifier 202A may further include a VoIP noise classifier 2028 for classifying the short-term segments as content-based VoIP noise or content-based non-VoIP noise, and the audio context classifier 204 may be configured to classify the short-term segments as contextual VoIP or contextual non-VoIP based on the confidence values ​​of VoIP speech, non-VoIP speech, VoIP noise and non-VoIP noise.

[0391] The content types VoIP speech, non-VoIP speech, VoIP noise and non-VoIP noise may be identified using existing techniques discussed in Part 6, Sections 1.2 and 7.1.

[0392] Alternatively, the audio content classifier 202A may further have a hierarchical structure as shown in Figure 35, in which the results from the speech / noise classifier 2025 are utilized to first classify short-term segments into speech or noise / background.

[0393] Based on the embodiment using the pure VoIP speech classifier 2026, if the short-term segment is determined to be speech by the speech / noise classifier 2025 (which in this situation is the pure speech classifier), the VoIP speech classifier 2026 proceeds to classify it as either VoIP speech or non-VoIP speech and calculates a binary classification result. Otherwise, the confidence value of VoIP speech may be considered low or the decision about VoIP speech may be uncertain.

[0394] Based on the embodiment using just VoIP noise classifier 2028, if the short-term segment is determined as noise by speech / noise classifier 2025 (which in this situation is just a noise (background) classifier), VoIP noise classifier 2028 proceeds to classify it as VoIP noise or non-VoIP noise and calculate a binary classification result. Otherwise, the confidence value of VoIP noise may be considered low or the decision about VoIP noise may be uncertain.

[0395] Here, since speech is generally an informational content type and noise / background is an interference content type, even if the short-term segment is not noise, the previous embodiment cannot definitively determine that the short-term segment is not contextual VoIP. If the short-term segment is not speech, it will probably not be contextual VoIP in the embodiment using only the VoIP speech classifier 2026. Therefore, in general, the embodiment using the teacher VoIP speech classifier 2026 can be implemented independently. Meanwhile, the other embodiment using the unit VoIP noise classifier 2028 can be used as a supplementary embodiment to cooperate with the embodiment using the VoIP speech classifier 2026, for example.

[0396] That is, both the VoIP speech classifier 2026 and the VoIP noise classifier 2028 may be used. If the short-term segment is determined to be speech by the speech / noise classifier 2025, the VoIP speech classifier 2026 proceeds to classify it as either VoIP speech or non-VoIP speech and calculates a binary classification result. If the short-term segment is determined to be noise by the speech / noise classifier 2025, the VoIP noise classifier 2028 proceeds to classify it as VoIP noise or non-VoIP noise and calculates a binary classification result. Otherwise, it may be considered that the short-term segment can be classified as non-VoIP.

[0397] The implementation of the speech / noise classifier 2025, the VoIP speech classifier 2026 and the VoIP noise classifier 2028 may employ any existing techniques and may be the audio content classifier 202 discussed in parts 1 to 6.

[0398] If the audio content classifier 202A implemented as described above does not ultimately classify a short-term segment as speech, noise or background, or as VoIP speech, non-VoIP speech, VoIP noise or non-VoIP noise, i.e., all associated confidence values ​​are low, then the audio content classifier 202A (and audio context classifier 204) may classify the short-term segment as non-VoIP.

[0399] To classify the short-term segments into VoIP or non-VoIP context types based on the results of the VoIP speech classifier 2026 and the VoIP noise classifier 2028, the audio context classifier 204 may employ machine learning based techniques as discussed in Section 7.1, or as a modification, more features may be used, including short-term features extracted directly from the short-term segments and / or the results of other audio content classifier(s) aimed at content types other than VoIP-related content types, as already discussed in Section 7.1.

[0400] Besides the above machine learning based techniques, an alternative approach to VoIP / non-VoIP classification can be heuristic rules that leverage domain knowledge and utilize classification results in the context of VoIP speech and VoIP noise.

[0401] If the current short-term segment at time t is determined as a VoIP utterance or a non-VoIP utterance, the classification result is taken directly as the VoIP / non-VoIP classification result, because the VoIP / non-VoIP utterance classification is robust as discussed above. That is, if the short-term segment is determined to be a VoIP utterance, it is a contextual VoIP. If the short-term segment is determined to be a non-VoIP utterance, it is a contextual non-VoIP.

[0402] When the VoIP speech classifier 2026 makes a binary decision regarding VoIP / non-VoIP speech with respect to the speech determined by the speech / noise classifier 2025 described above, the confidence values ​​of the VoIP and non-VoIP speech may be complementary, i.e., their sum is 1 (where 0 represents 100% no and 1 represents 100% positive), and the confidence value thresholds for distinguishing between VoIP and non-VoIP speech may actually point to the same point. If the VoIP speech classifier 2026 is not a binary classifier, the confidence values ​​of the VoIP and non-VoIP speech may not be complementary, and the confidence value thresholds for distinguishing between VoIP and non-VoIP speech may not necessarily point to the same point.

[0403] However, when the VoIP or non-VoIP speech confidence is close to a threshold and fluctuates around the threshold, the VoIP / non-VoIP classification result may switch too frequently. To avoid such fluctuation, a buffering scheme may be provided. The thresholds for both VoIP and non-VoIP speech may be set larger, so that it is not so easy to switch from the current content type to the other content type. For ease of description, the confidence value for non-VoIP speech may be converted to the confidence value for VoIP speech. That is, if the confidence value is high, the short-term segment is considered to be closer to VoIP speech, and if the confidence value is low, the short-term segment is considered to be closer to non-VoIP speech. Although for the non-binary classifier as above, a high confidence value for non-VoIP speech does not necessarily mean a low confidence value for VoIP speech, such a simplification can well reflect the essence of the present solution, and the related claims written using the language of a binary classifier are interpreted to cover the equivalent solution for a non-binary classifier.

[0404] The buffering scheme is shown in Figure 36. There is a buffer region between two thresholds Th1 and Th2 (Th1 >= Th2). When the confidence value v(t) of a VoIP utterance falls in this region, the context classification does not change. This is shown by the left and right arrows in Figure 36. Only when the confidence value v(t) is greater than the larger threshold Th1, the short-term segment will be classified as VoIP (as shown by the arrow at the bottom of Figure 36), and only when the confidence value is not greater than the smaller threshold Th2, the short-term segment will be classified as non-VoIP (as shown by the arrow at the top of Figure 36).

[0405] The situation is similar if a VoIP noise classifier 2028 is used instead. To make the solution more robust, the VoIP speech classifier 2026 and the VoIP noise classifier 2028 may be used jointly. The audio context classifier 204A may then be configured to: classify the short-term segment as contextual VoIP if the confidence value of the VoIP speech is greater than a first threshold or the confidence value of the VoIP noise is greater than a third threshold; classify the short-term segment as contextual non-VoIP if the confidence value of the VoIP speech is not greater than a second threshold not greater than said first threshold or the confidence value of the VoIP noise is not greater than a fourth threshold not greater than said third threshold; otherwise classify the short-term segment as contextual for the last short-term segment.

[0406] Here, the first threshold may be equal to the second threshold, and the third threshold may be equal to the fourth threshold. This is especially true for, but not limited to, the binary VoIP speech classifier and the binary VoIP noise classifier. However, in general, the third and fourth thresholds should not be equal to each other, since the VoIP noise classification result is not so robust. Both should be far from 0.5 (0 indicates high confidence that it is non-VoIP noise, and 1 indicates high confidence that it is VoIP noise).

[0407] <Section 7.3 Smoothing Fluctuation> To avoid rapid fluctuations, another solution is to smooth the confidence values ​​determined by the audio content classifier. Thus, a type smoothing unit 203A may be included in the audio classifier 200A, as shown in Figure 37. For the confidence values ​​of each of the four VoIP-related content types discussed above, the smoothing scheme discussed in Section 1.3 may be employed.

[0408] Alternatively, similar to Section 7.2, VoIP speech and non-VoIP speech may be considered as a pair with complementary confidence values. VoIP noise and non-VoIP noise may also be considered as a pair with complementary confidence values. In such a situation, only one of each pair needs to be smoothed, and the smoothing scheme discussed in Section 1.3 may be employed.

[0409] Taking the confidence value of a VoIP utterance as an example, formula (3) may be rewritten as follows:

[0410] v(t)=β·v(t-1)+(1-β)·voipSpeechConf(t) (3") Here, v(t) is the smoothed VoIP speech confidence value at time t, v(t-1) is the smoothed VoIP speech confidence value at the last time point, voipSpeechConf is the VoIP speech confidence at the current time t before smoothing, and α is a weighting coefficient.

[0411] In one variation, if there is a speech / noise classifier 2025 as described above, then if the speech confidence value for a short-term segment is low, then the short-term segment cannot be robustly classified as VoIP speech, and one can directly set voipSeechConf(t)=v(t−1) without actually running the VoIP speech classifier 2026.

[0412] Alternatively, in the above situation, one can set voipSpeechConf(t)=0.5 (or any other value not higher than 0.5, such as 0.4-0.5) to indicate the uncertain case (where confidence=1 indicates high confidence that it is VoIP and confidence=0 indicates high confidence that it is not VoIP).

[0413] Thus, according to this variant, as shown in Fig. 37, the audio content classifier 200A may further comprise a speech / noise classifier 2025 for identifying the content type of the speech of the short-term segment, and the type smoothing unit 203A may be configured to set the confidence value of the VoIP speech for the current short-term segment before smoothing as a predefined confidence value (such as 0.5 or other value such as 0.4-0.5) or as the smoothed confidence value of the last short-term segment whose confidence value for the content type speech classified by the speech / noise classifier is lower than the fifth threshold. In such a situation, the VoIP speech classifier 2026 may or may not act. Alternatively, the setting of the confidence value may be performed by the VoIP speech classifier 2026. This is equivalent to a solution in which this action is performed by the type smoothing unit 203A, and the claims should be interpreted to cover both situations. Furthermore, although the language used herein is that "the confidence value for the content type speech as classified by the speech / noise classifier is lower than the fifth threshold", the scope of protection is not so limited and is equivalent to situations where a short-term segment is classified as a content type other than speech.

[0414] For the VoIP noise confidence value, the situation is similar and a detailed explanation is omitted here.

[0415] To avoid sudden fluctuations, yet another solution is to smooth the confidence values ​​determined by the audio context classifier 204A, where the smoothing scheme discussed in section 1.3 may be adopted.

[0416] To avoid the sudden fluctuation, yet another solution is to delay the transition of the context type between VoIP and non-VoIP, and the same method as described in section 1.6 may be used. As described in section 1.6, the timer 916 may be external to the audio classifier or may be internal as part of the audio classifier. Thus, as shown in FIG. 38, the audio classifier 200A may further include a timer 916. Then, the audio classifier is configured to continue outputting the current context type (a context type is an instance of an audio type) until the duration length of the new context type reaches a sixth threshold. By referring to section 1.6, a detailed description may be omitted here.

[0417] Additionally or alternatively, as another way to delay the transition between VoIP and non-VoIP, the first and / or second thresholds mentioned above for VoIP / non-VoIP classification may be different depending on the context type of the last short-term segment. That is, the first and / or second thresholds are larger when the context type of the new short-term segment is different from the context type of the last short-term segment, whereas the first and / or second thresholds are smaller when the context type of the new short-term segment is the same as the context type of the last short-term segment. In this way, the context type tends to be maintained at the current context type, and thus the sudden fluctuation of the context type may be suppressed to some extent.

[0418] <Section 7.4 Combinations of embodiments and application scenarios> As with Part 1, all of the embodiments and variations thereof discussed above may be implemented in any combination thereof, and any components mentioned in different parts / embodiments but having the same or similar functionality may be implemented as the same or separate components.

[0419] For example, any two or more of the solutions described in sections 7.1 to 7.3 may be combined with each other, and any of these combinations may further be combined with any of the embodiments described or implied in parts 1 to 6. In particular, the embodiments discussed in this part and any combinations thereof may be combined with the audio processing device / method or volume leveller / control method embodiments discussed in part 4.

[0420] <Section 7.5 VoIP classification method> As in the first part, it is clear that in the course of describing the audio classifier in the above embodiment, several processes or methods are also disclosed. In the following, an overview of these methods is given without repeating some of the details already discussed above.

[0421] In one embodiment shown in FIG. 39, an audio classification method includes identifying a content type of a short-term segment of an audio signal (operation 4004) and then identifying a context type of the short-term segment based at least on the identified content type (operation 4008).

[0422] In order to dynamically and quickly identify the context type of an audio signal, the audio classification method in this part is particularly useful in identifying contextual VoIP and non-VoIP. In such a situation, the short-term segment may be first classified as a content-type VoIP speech or a content-type non-VoIP speech, and the above operation of identifying the context type is configured to classify the short-term segment as a contextual VoIP or a contextual non-VoIP based on the confidence values ​​of the VoIP speech and the non-VoIP speech.

[0423] Alternatively, the short-term segments may first be classified as content-type VoIP noise or content-type non-VoIP noise, and the above operation of identifying the context type may be configured to classify the short-term segments as context-type VoIP or context-type non-VoIP based on the confidence values ​​of the VoIP noise and non-VoIP noise.

[0424] Speech and noise may be considered jointly. In such a situation, the above operation of identifying a context type may be configured to classify a short-term segment as contextual VoIP or contextual non-VoIP based on the confidence values ​​of VoIP speech, non-VoIP speech, VoIP noise and non-VoIP noise.

[0425] To identify the context type of a short-term segment, a machine learning model may be used, incorporating both the confidence value of the content type of the short-term segment and other features extracted from the short-term segment as features.

[0426] The above operation of identifying the context type may be realized based on a heuristic rule. When only VoIP and non-VoIP utterances are involved, the heuristic rule is as follows: classify the short-term segment as context type VoIP if the confidence value of the VoIP utterance is greater than a first threshold; classify the short-term segment as context type non-VoIP if the confidence value of the VoIP utterance is not greater than a second threshold, where the second threshold is not greater than the first threshold; otherwise, classify the short-term segment as context type for the last short-term segment.

[0427] The heuristic rules for situations involving only VoIP noise and non-VoIP noise are similar.

[0428] When both speech and noise are involved, the heuristic rules are: classify the short-term segment as contextual VoIP if the confidence value of the VoIP speech is greater than a first threshold or the confidence value of the VoIP noise is greater than a third threshold; classify the short-term segment as contextual non-VoIP if the confidence value of the VoIP speech is not greater than a second threshold not greater than the first threshold or the confidence value of the VoIP noise is not greater than a fourth threshold not greater than the third threshold; otherwise classify the short-term segment as contextual for the last short-term segment.

[0429] The smoothing scheme discussed in Section 1.3 and Section 1.8 may be adopted here and will not be described in detail. As a modification to the smoothing scheme described in Section 1.3, before the smoothing operation 4106, the method may further include identifying content-type speech from short-term segments (operation 40040 in FIG. 40). Here, if the confidence value for the content-type speech is lower than the fifth threshold ("No" in operation 40041), the confidence value of the VoIP speech for the current short-term segment before smoothing is set as a predetermined confidence value or the smoothed confidence value of the last short-term segment (operation 40044 in FIG. 40).

[0430] Otherwise, if the content-based speech identification operation robustly determines the short-term segment as speech ("Yes" at operation 40041), the short-term segment is further classified as VoIP speech or non-VoIP speech (operation 40042) prior to the smoothing operation 4106.

[0431] In fact, even without using the smoothing scheme, the method may first identify content types speech and / or noise, and when the short-term segments are classified as speech or noise, a further classification is implemented to classify the short-term segments as one of VoIP speech and non-VoIP speech or one of VoIP noise or non-VoIP noise. Then, an operation of identifying the context type is performed.

[0432] As noted in sections 1.6 and 1.8, the transition schemes discussed therein may be incorporated as part of the audio classification method described herein and will not be discussed in detail here. Briefly, the method may further include measuring the duration that the act of identifying the context type successively outputs the same context type. The audio classification method is configured to continue outputting the current context type until the length of time that the new context type lasts reaches a sixth threshold.

[0433] Similarly, different sixth thresholds may be set for different transition pairs from one context type to another, and further, the sixth threshold may be negatively correlated with the confidence value of the new context type.

[0434] As a modification to the transition scheme in the audio classification method specifically directed to VoIP / non-VoIP classification, any one or more of the first through fourth thresholds for the current short-term segment may be set differently depending on the context type of the last short-term segment.

[0435] As with the audio processing device embodiments, any combination of the audio processing method embodiments and variants thereof is feasible. On the other hand, any aspect of the audio processing method embodiments and variants thereof may be separate solutions. Furthermore, any two or more solutions described in this section may be combined with each other, and these combinations may further be combined with any embodiment described or implied in other parts of the present disclosure. In particular, the audio classification method described herein may be used in the audio processing methods described above, in particular the volume leveller control method.

[0436] As discussed in the introduction to the "Description of the Preferred Embodiments" section of this application, embodiments of the present application may be embodied in hardware or software or both. Figure 41 is a block diagram illustrating an exemplary system for implementing aspects of the present application.

[0437] 41, a central processing unit (CPU) 4201 executes various processes according to a program stored in a read-only memory (ROM) 4202 or a program loaded from a storage section 4208 into a random access memory (RAM) 4203. In the RAM 4203, data required when the CPU 4201 executes the various processes is also stored as necessary.

[0438] The CPU 4201, the ROM 4202, and the RAM 4203 are connected to one another via a bus 4204. An input / output interface 4205 is also connected to the bus 4204.

[0439] The following components are connected to the input / output interface 4205: an input section 4206 including a keyboard, a mouse, etc.; an output section 4207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a loudspeaker, etc.; a storage section 4208 including a hard disk, etc.; and a communication section 4209 including a network interface card such as a LAN card, a modem, etc. The communication section 4209 performs communication processes over a network such as the Internet.

[0440] A drive 4210 is also connected to the input / output interface 4205 as necessary. A removable medium 4211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted in the drive 4210 as necessary. Thereby, a computer program read from there is installed in the storage unit 4208 as necessary.

[0441] When the above components are implemented by software, the program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 4211 .

[0442] It should be noted that the terms used in this application are merely for the purpose of describing specific embodiments and are not intended to limit the application. As used herein, the singular forms are intended to include the plural unless the context clearly indicates otherwise. Furthermore, the terms "comprise" and / or "have" as used herein indicate the presence of stated features, integers, operations, steps, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, operations, steps, elements, components, and / or groups thereof.

[0443] Corresponding structures, materials, steps, and equivalents of any means or operation-added element in a claim are intended to include any structure, material, or step for performing a function in combination with other claimed elements explicitly recited in the claim. The description of the present application has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the application in the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present application. The embodiments have been chosen and described in order to best explain the principles and practical application of the present application and to enable those skilled in the art to understand the present application in various embodiments with various modifications suitable for the specific use contemplated.

[0444] Several aspects will be described. [Aspect 1] with an audio content classifier for identifying the content type of an audio signal in real time; and an adjusting unit for adjusting the volume leveller in a continuous manner based on the identified content type, the adjusting unit is configured to positively correlate a dynamic gain of the volume leveller with an informational content type of the audio signal and to negatively correlate a dynamic gain of the volume leveller with an interfering content type of the audio signal. Volume leveller controller. [Aspect 2] 2. The volume leveler controller of embodiment 1, wherein the content type of the audio signal includes one of speech, short-term music, noise, and background sounds. [Aspect 3] 2. The volume leveller controller of embodiment 1, wherein noise is considered to be a coherent content type. Aspect 4 2. The volume leveler controller of embodiment 1, wherein the adjusting unit is configured to adjust a dynamic gain of the volume leveler based on a confidence value of the content type. Aspect 5 5. The volume leveler controller of embodiment 4, wherein the adjusting unit is configured to adjust the dynamic gain via a transfer function of a confidence value of the content type. Aspect 6 The volume leveler controller of aspect 1, wherein the audio content classifier is configured to classify the audio signal into a plurality of content types having corresponding confidence values, and the adjustment unit is configured to take into account at least some of the plurality of audio types through weighting the confidence values ​​of the plurality of content types based on the importance of the plurality of content types. Aspect 7 2. The volume leveler controller of claim 1, wherein the audio content classifier is configured to classify the audio signal into a plurality of content types having corresponding confidence values, and the adjustment unit is configured to modify a weight of a content type using the confidence value of at least one other content type. Aspect 8 2. The volume leveler controller of claim 1, wherein the audio content classifier is configured to classify the audio signal into a plurality of content types having corresponding confidence values, and the adjustment unit is configured to take into account at least some of the plurality of audio types by weighting an effect of the plurality of content types based on the confidence values. Aspect 9 9. The volume leveler controller of embodiment 8, wherein the adjustment unit is configured to consider at least one prevalent content type based on the confidence value. Aspect 10 The volume leveler controller of embodiment 1, wherein the audio content classifier is configured to classify the audio signal into a plurality of coherent content types and / or a plurality of informational content types having corresponding confidence values, and the adjustment unit is configured to take into account at least one predominant coherent content type and / or at least one predominant informational content type based on the confidence values. Aspect 11 11. The volume leveller controller of any one of aspects 1 to 10, further comprising a type smoothing unit for smoothing, for each content type, a confidence value of the audio signal at a current point in time based on past confidence values ​​of the audio signal. Aspect 12 The volume leveller controller of claim 11, wherein the smoothing unit is configured to determine a smoothed confidence value of the audio signal at a current time by calculating a weighted sum of a current actual confidence value and a smoothed confidence value at a last time point. Aspect 13 11. The volume leveler controller of any one of aspects 1 to 10, further comprising an audio context classifier that identifies a context type of the audio signal, and the adjustment unit is configured to adjust the dynamic gain range based on a confidence value of the context type. Aspect 14 11. The volume leveler controller of any one of aspects 1 to 10, further comprising an audio context classifier that identifies a context type of the audio signal, and the adjustment unit is configured to classify the content type of the audio signal as informative or interfering based on the context type of the audio signal. Aspect 15 15. The volume leveler controller of embodiment 14, wherein the context type of the audio signal comprises one of VoIP, cinematic media, long-term music, and gaming. Aspect 16 The volume leveller controller of embodiment 14, wherein in a contextual VoIP audio signal, background sounds are considered to be an interfering content type, whereas in a contextual non-VoIP audio signal, background sounds and / or speech and / or music are considered to be informational content types. Aspect 17 15. The volume leveler controller of embodiment 14, wherein the context type of the audio signal comprises high quality audio or low quality audio. Aspect 18 15. The volume leveler controller of embodiment 14, wherein content types in an audio signal of different context types are assigned different weights depending on the context type of the audio signal. Aspect 19 The volume leveler controller of embodiment 14, wherein the audio context classifier is configured to classify the audio signal into a plurality of context types having corresponding confidence values, and the adjustment unit is configured to take into account at least some of the plurality of context types through weighting the confidence values ​​of the plurality of context types based on the importance of the plurality of context types. Aspect 20 15. The volume leveler controller of claim 14, wherein the audio context classifier is configured to classify the audio signal into a plurality of context types having corresponding confidence values, and the adjustment unit is configured to take into account at least some of the plurality of context types by weighting the effects of the plurality of context types based on the confidence values. Aspect 21 the audio content classifier is configured to identify the content type based on short-term segments of the audio signal; 15. The volume leveler controller of claim 14, wherein the audio context classifier is configured to identify the context type based at least in part on short-term segments of the audio signal based on a content type identified by the audio content classifier. Aspect 22 the audio content classifier comprises a VoIP speech classifier for classifying short-term segments as content-based VoIP speech or content-based non-VoIP speech; 22. The volume leveler controller of claim 21, wherein the audio context classifier is configured to classify the short-term segments as contextual VoIP or contextual non-VoIP based on confidence values ​​of VoIP speech and non-VoIP speech. Aspect 23 The audio content classifier further comprises: a VoIP noise classifier for classifying the short-term segments into a VoIP noise content type and a non-VoIP noise content type; the audio context classifier is configured to classify the short-term segments as contextual VoIP or contextual non-VoIP based on confidence values ​​of VoIP speech, non-VoIP speech, VoIP noise, and non-VoIP noise; 23. The volume leveller controller of embodiment 22. Aspect 24 2. The audio context classifier: The volume leveller controller of aspect 22 is configured to classify the short-term segment as contextual VoIP if the confidence value of the VoIP speech is greater than a first threshold; classify the short-term segment as contextual non-VoIP if the confidence value of the VoIP speech is not greater than a second threshold that is not greater than the first threshold; and otherwise classify the short-term segment as contextual for the last short-term segment. Aspect 25 2. The audio context classifier: The volume leveller controller of aspect 23 is configured to classify the short-term segment as contextual VoIP if the confidence value of VoIP speech is greater than a first threshold or the confidence value of VoIP noise is greater than a third threshold; classify the short-term segment as contextual non-VoIP if the confidence value of VoIP speech is not greater than a second threshold that is not greater than the first threshold or the confidence value of VoIP noise is not greater than a fourth threshold that is not greater than the third threshold; and otherwise classify the short-term segment as contextual for the last short-term segment. Aspect 26 26. The volume leveler controller of any one of aspects 21 to 25, further comprising a type smoothing unit for smoothing a confidence value of the content type at a current time based on a past confidence value of the content type. Aspect 27 The volume smoother controller of embodiment 26, wherein the smoothing unit is configured to determine a smoothed confidence value of a current short-term segment by calculating a weighted sum of the confidence value of the current short-term segment and a smoothed confidence value of the last short-term segment. Aspect 28 27. The volume leveler controller of claim 26, wherein the audio content classifier further comprises a speech / noise classifier that identifies a content type of speech in the short-term segment, and the type smoothing unit is configured to set the confidence value of VoIP speech for the current short-term segment before smoothing as a predetermined confidence value or as the smoothed confidence value of the last short-term segment in which the confidence value for the content type speech classified by the speech / noise classifier is lower than a fifth threshold. Aspect 29 24. The volume leveler controller of claim 22 or 23, wherein the audio context classifier is configured to classify the short-term segments based on a machine learning model using as features a confidence value of a content type of the short-term segments and other features extracted from the short-term segments. Aspect 30 30. The volume leveler controller of any one of aspects 14 to 29, further comprising a timer that measures the duration that the audio context classifier continuously outputs the same context type, and the adjustment unit is configured to continue using the current context type until the duration length of the new context type reaches a sixth threshold. Aspect 31 31. The volume leveler controller of embodiment 30, wherein different sixth thresholds are set for different pairs of transitions from one context type to another context type. Aspect 32 31. The volume leveler controller of embodiment 30, wherein the sixth threshold is negatively correlated with a confidence value of the new context type. Aspect 33 26. The volume leveller controller of embodiment 24 or 25, wherein the first and / or second thresholds vary depending on a context type of the last short-term segment. Aspect 34 34. An audio processing device comprising the volume leveller controller of any one of aspects 1 to 33. Aspect 35 an audio content classifier for identifying the content type of short-term segments of an audio signal; and an audio context classifier that identifies a context type for the short-term segments based at least in part on a content type identified by the audio content classifier. Audio classifier. Aspect 36 the audio content classifier comprises a VoIP speech classifier for classifying the short-term segments as content-based VoIP speech or content-based non-VoIP speech; the audio context classifier is configured to classify the short-term segments as contextual VoIP or contextual non-VoIP based on confidence values ​​of VoIP speech and non-VoIP speech. 36. The audio classifier of embodiment 35. Aspect 37 The audio content classifier further comprises: a VoIP noise classifier for classifying the short-term segments into content-based VoIP noise and content-based non-VoIP noise; the audio context classifier is configured to classify the short-term segments as contextual VoIP or contextual non-VoIP based on confidence values ​​of VoIP speech, non-VoIP speech, VoIP noise, and non-VoIP noise; 37. The audio classifier of embodiment 36. Aspect 38 2. The audio context classifier: 38. The audio classifier of claim 37, configured to classify the short-term segment as contextual VoIP if the confidence value of the VoIP utterance is greater than a first threshold; classify the short-term segment as contextual non-VoIP if the confidence value of the VoIP utterance is not greater than a second threshold that is not greater than the first threshold; and otherwise classify the short-term segment as contextual for the last short-term segment. Aspect 39 2. The audio context classifier: classifying the short-term segment as contextual VoIP if the confidence value of VoIP speech is greater than a first threshold or the confidence value of VoIP noise is greater than a third threshold; classifying the short-term segment as contextual non-VoIP if the confidence value of VoIP speech is not greater than a second threshold not greater than the first threshold or the confidence value of VoIP noise is not greater than a fourth threshold not greater than the third threshold; otherwise classifying the short-term segment as contextual for the last short-term segment. 38. The audio classifier of embodiment 37. Aspect 40 40. The audio classifier of any one of aspects 35-39, further comprising a type smoothing unit for smoothing a current confidence value of the content type based on past confidence values ​​of the content type. Aspect 41 The audio classifier of embodiment 40, wherein the model smoothing unit is configured to determine a smoothed confidence value of the current short-term segment by calculating a weighted sum of the confidence value of the current short-term segment and a smoothed confidence value of the last short-term segment. Aspect 42 42. The audio classifier of claim 41, wherein the audio content classifier further comprises a speech / noise classifier for identifying content-type speech from the short-term segments, and the type smoothing unit is configured to set the confidence value of VoIP speech for the current short-term segment before smoothing as a predetermined confidence value or as a smoothed confidence value of the last short-term segment in which the confidence value for content-type speech classified by the speech / noise classifier is lower than a fifth threshold. Aspect 43 38. The audio classifier of claim 36 or 37, wherein the audio context classifier is configured to classify the short-term segment based on a machine learning model using a confidence value of a content type of the short-term segment and other features extracted from the short-term segment as features. Aspect 44 40. The audio classifier of claim 38 or 39, further comprising a timer that measures the duration that the audio context classifier continuously outputs the same context type, and the audio classifier is configured to continue outputting the current context type until the duration length of the new context type reaches a sixth threshold. Aspect 45 45. The audio classifier of embodiment 44, wherein different sixth thresholds are set for different pairs of transitions from one context type to another context type. Aspect 46 45. The audio classifier of embodiment 44, wherein the sixth threshold is negatively correlated with the confidence value of the new context type. Aspect 47 40. The audio classifier of embodiment 38 or 39, wherein the first and / or second thresholds vary depending on a context type of the last short-term segment. Aspect 48 48. An audio processing device having an audio classifier according to any one of aspects 35 to 47. Aspect 49 identifying a content type of the audio signal in real time; adjusting a volume leveller in a continuous manner based on the identified content type by positively correlating a dynamic gain of the volume leveller with an informational content type of the audio signal and negatively correlating a dynamic gain of the volume leveller with an interfering content type of the audio signal. Volume leveller control method. Aspect 50 50. The volume leveler control method of embodiment 49, wherein the content type of the audio signal includes one of speech, short-term music, noise, and background sounds. Aspect 51 50. The volume leveller control method of embodiment 49, wherein noise is considered to be a coherent content type. Aspect 52 50. The volume leveler control method of embodiment 49, wherein the adjusting operation is configured to adjust a dynamic gain of the volume leveler based on a confidence value of the content type. Aspect 53 53. The volume leveler control method of embodiment 52, wherein the adjusting operation is configured to adjust the dynamic gain via a transfer function of a confidence value of the content type. Aspect 54 50. The volume leveler control method of claim 49, wherein the audio signal is classified into a plurality of content types having corresponding confidence values, and the adjusting operation is configured to take into account at least some of the plurality of audio types through weighting the confidence values ​​of the plurality of content types based on the importance of the plurality of content types. Aspect 55 50. The volume leveler control method of embodiment 49, wherein the audio signal is classified into a plurality of content types having corresponding confidence values, and the adjusting operation is configured to modify the weight of a content type using the confidence value of at least one other content type. Aspect 56 50. The volume leveler control method of embodiment 49, wherein the audio signal is classified into a plurality of content types having corresponding confidence values, and the adjusting operation is configured to take into account at least some of the plurality of content types by weighting the effects of the plurality of content types based on the confidence values. Aspect 57 57. The volume leveler control method of embodiment 56, wherein the adjusting operation is configured to take into account at least one prevalent content type based on the trust value. Aspect 58 The volume leveler control method of embodiment 56, wherein the audio signal is classified into a plurality of coherent content types and / or a plurality of informational content types with corresponding confidence values, and the adjusting operation is configured to take into account at least one predominant coherent content type and / or at least one predominant informational content type based on the confidence values. Aspect 59 59. The volume leveller control method of any one of aspects 49 to 58, further comprising, for each content type, smoothing a confidence value of the audio signal at a current time point based on past confidence values ​​of the audio signal. Aspect 60 The volume leveler control method of aspect 59, wherein the type smoothing operation is configured to determine a smoothed confidence value of the audio signal at a current time by calculating a weighted sum of a current actual confidence value and a smoothed confidence value at a last time point. Aspect 61 59. The volume leveler control method of any one of aspects 49 to 58, further comprising a step of identifying a context type of the audio signal, and the adjusting operation is configured to adjust the dynamic gain range based on a confidence value of the context type. Aspect 62 A volume leveler control method as described in any one of aspects 49 to 58, further comprising a step of identifying a context type of the audio signal, and the adjusting operation is configured to consider the content type of the audio signal as informative or interfering based on the context type of the audio signal. Aspect 63 63. The volume leveler control method of embodiment 62, wherein the context type of the audio signal includes one of VoIP, cinematic media, long-term music, and gaming. Aspect 64 A volume leveller control method as described in embodiment 62, wherein in a contextual VoIP audio signal, background sounds are considered to be an interfering content type, while in a contextual non-VoIP audio signal, background sounds and / or speech and / or music are considered to be informational content types. Aspect 65 63. The volume leveler control method of embodiment 62, wherein the context type of the audio signal includes high quality audio or low quality audio. Aspect 66 63. The volume leveler control method of embodiment 62, wherein content types in an audio signal of different context types are assigned different weights depending on the context type of the audio signal. Aspect 67 63. The volume leveler control method of claim 62, wherein the audio signal is classified into a plurality of context types having corresponding confidence values, and the adjusting operation is configured to take into account at least some of the plurality of context types through weighting the confidence values ​​of the plurality of context types based on the importance of the plurality of context types. Aspect 68 63. The volume leveler control method of claim 62, wherein the audio signal is classified into a plurality of context types having corresponding confidence values, and the adjusting operation is configured to take into account at least some of the plurality of context types by weighting the effects of the plurality of context types based on the confidence values. Aspect 69 the operation of identifying a content type is configured to identify the content type based on short-term segments of the audio signal; and wherein the operation of identifying the context type is configured to identify the context type based on short-term segments of the audio signal that are based at least in part on an identified content type. A volume leveller control method as described in embodiment 62. Aspect 70 the act of identifying a content type includes classifying the short-term segments as content-type VoIP speech or content-type non-VoIP speech; 70. The volume leveler control method of embodiment 69, wherein the operation of identifying a context type is configured to classify the short-term segment as contextual VoIP or contextual non-VoIP based on confidence values ​​of VoIP speech and non-VoIP speech. Aspect 71 The act of identifying the content type further comprises: classifying the short-term segments into content-based VoIP noise and content-based non-VoIP noise; and the operation of identifying a context type is configured to classify the short-term segment as contextual VoIP or contextual non-VoIP based on confidence values ​​of VoIP speech, non-VoIP speech, VoIP noise, and non-VoIP noise. A volume leveller control method according to embodiment 70. Aspect 72 The behavior of identifying the context type is: classifying the short-term segment as contextual VoIP if the confidence value of the VoIP utterance is greater than a first threshold; classifying the short-term segment as contextual non-VoIP if a confidence value of a VoIP utterance is not greater than a second threshold not greater than the first threshold; otherwise, classifying the short-term segment as a context type for a last short-term segment. A volume leveller control method according to embodiment 70. Aspect 73 The context-identifying actions are: classifying the short-term segment as contextual VoIP if a confidence value of VoIP speech is greater than a first threshold or a confidence value of VoIP noise is greater than a third threshold; classifying the short-term segment as contextual non-VoIP if a confidence value of VoIP speech is not greater than a second threshold not greater than the first threshold or a confidence value of VoIP noise is not greater than a fourth threshold not greater than the third threshold; 72. The volume leveler control method of embodiment 71, further comprising: otherwise classifying the short-term segment as a context type for a last short-term segment. Aspect 74 74. The volume leveler control method of any one of aspects 69 to 73, further comprising smoothing the current trust value of the content type based on past trust values ​​of the content type. Aspect 75 The volume smoother control method of embodiment 74, wherein the type smoothing operation is configured to determine a smoothed confidence value of the current short-term segment by calculating a weighted sum of the confidence value of the current short-term segment and the smoothed confidence value of the last short-term segment. Aspect 76 76. The volume leveler control method of claim 75, further comprising identifying a content type of speech of the short-term segment, wherein the confidence value of VoIP speech for the current short-term segment before smoothing is set as a predetermined confidence value or as the smoothed confidence value of the last short-term segment whose confidence value for content-type speech is lower than a fifth threshold. Aspect 77 The volume leveler control method of aspect 70 or 71, wherein the short-term segments are classified based on a machine learning model using, as a characteristic, a confidence value of the content type of the short-term segments and other features extracted from the short-term segments. Aspect 78 78. The volume leveler control method of any one of aspects 62 to 77, wherein the act of identifying a context type further includes measuring the duration of continuously outputting the same context type, and the act of adjusting is configured to continue using the current context type until the duration length of the new context type reaches a sixth threshold. Aspect 79 80. The volume leveler control method of embodiment 78, wherein different sixth thresholds are set for different pairs of transitions from one context type to another context type. Aspect 80 80. The volume leveler control method of embodiment 78, wherein the sixth threshold is negatively correlated with the confidence value of the new context type. Aspect 81 74. The volume leveller control method of embodiment 72 or 73, wherein the first and / or second thresholds vary depending on a context type of the last short-term segment. Aspect 82 identifying a content type of a short-term segment of the audio signal; and identifying a context type for the short-term segment based at least in part on the identified content type. Audio classification methods. Aspect 83 the act of classifying a content type includes classifying the short-term segments as content-type VoIP speech or content-type non-VoIP speech; and the operation of identifying a context type is configured to classify the short-term segment as contextual VoIP or contextual non-VoIP based on confidence values ​​of VoIP utterances and non-VoIP utterances. 83. The audio classification method of embodiment 82. Aspect 84 The content type classification behavior is further improved by classifying the short-term segments into content-based VoIP noise or content-based non-VoIP noise; and the operation of identifying a context type is configured to classify the short-term segment as contextual VoIP or contextual non-VoIP based on confidence values ​​of VoIP speech, non-VoIP speech, VoIP noise, and non-VoIP noise. 84. The audio classification method of embodiment 83. Aspect 85 The behavior of identifying the context type is: classifying the short-term segment as contextual VoIP if the confidence value of the VoIP utterance is greater than a first threshold; classifying the short-term segment as contextual non-VoIP if a confidence value of a VoIP utterance is not greater than a second threshold not greater than the first threshold; otherwise, classifying the short-term segment as a context type for a last short-term segment. 84. The audio classification method of embodiment 83. Aspect 86 The behavior of identifying the context type is: classifying the short-term segment as contextual VoIP if a confidence value of VoIP speech is greater than a first threshold or a confidence value of VoIP noise is greater than a third threshold; classifying the short-term segment as contextual non-VoIP if a confidence value of VoIP speech is not greater than a second threshold not greater than the first threshold or a confidence value of VoIP noise is not greater than a fourth threshold not greater than the third threshold; otherwise, classifying the short-term segment as a context type for a last short-term segment. 85. The audio classification method of embodiment 84. Aspect 87 87. The method of audio classification of any one of aspects 82 to 86, further comprising smoothing a current confidence value of the content type based on past confidence values ​​of the content type. Aspect 88 The audio classification method of embodiment 87, wherein the operation of smoothing the shape is configured to determine a smoothed confidence value of the current short-term segment by calculating a weighted sum of the confidence value of the current short-term segment and the smoothed confidence value of the last short-term segment. Aspect 89 90. The audio classification method of embodiment 88, further comprising identifying content-type speech from the short-term segments, wherein a confidence value of VoIP speech for the current short-term segment before smoothing is set as a predetermined confidence value or as the smoothed confidence value of the last short-term segment in which the confidence value for content-type speech is lower than a fifth threshold. Aspect 90 85. The audio classification method of claim 83 or 84, wherein the operation of identifying a context type is configured to classify the short-term segment based on a machine learning model using a confidence value of the content type of the short-term segment and other features extracted from the short-term segment as features. Aspect 91 87. The audio classification method of embodiment 85 or 86, wherein the operation of identifying a context type further includes a step of measuring the duration for which the same context type is continuously output, and the audio classification method is configured to continue outputting the current context type until the length of time for which the new context type lasts reaches a sixth threshold. Aspect 92 92. The audio classification method of embodiment 91, wherein different sixth thresholds are set for different pairs of transitions from one context type to another context type. Aspect 93 92. The audio classification method of embodiment 91, wherein the sixth threshold is negatively correlated with the confidence value of the new context type. Aspect 94 87. The audio classification method of embodiment 85 or 86, wherein the first and / or second thresholds vary depending on a context type of the last short-term segment. Aspect 95 1. A computer-readable medium having computer program instructions recorded thereon that, when executed by a processor, cause the processor to perform a volume leveler control method, the volume leveler control method comprising: identifying a content type of the audio signal in real time; adjusting a volume leveller in a continuous manner based on the identified content type by positively correlating a dynamic gain of the volume leveller with an informational content type of the audio signal and negatively correlating a dynamic gain of the volume leveller with an interfering content type of the audio signal. Computer-readable medium. Aspect 96 1. A computer readable medium having computer program instructions recorded thereon that, when executed by a processor, cause the processor to perform an audio classification method, the audio classification method comprising: identifying a content type of a short-term segment of the audio signal; and identifying a context type for the short-term segment based at least in part on the identified content type. Computer-readable medium.

Claims

1. 1. A method for loudness normalization based on a target loudness value, the method comprising: determining dynamic gain parameters to be applied to audio frames of an audio signal based on short-term or long-term characteristics of the audio signal, said determining comprising determining at least one dynamic gain parameter for a first audio frame based on the short-term characteristics of the audio signal and the target loudness value, and determining at least one dynamic gain parameter for a second audio frame based on the long-term characteristics of the audio signal and the target loudness value; scaling at least a portion of the first audio frame by the at least one dynamic gain parameter for the first audio frame to modify the loudness of the first audio frame to match the target loudness upon playback; and scaling at least a portion of the second audio frame by the at least one dynamic gain parameter for the second audio frame to modify the loudness of the second audio frame to match the target loudness upon playback; the long-term characteristics are determined differently than the short-term characteristics; Loudness normalization method.

2. The loudness normalization method of claim 1 , wherein the dynamic gain parameter is identified and applied in real time.

3. 2. The loudness normalization method of claim 1, in which dialogue enhancements are applied that have the effect of making dialogue more noticeable within a particular context.

4. 2. A method for loudness normalization according to claim 1, in which loudness equalization is applied having an effect on one or more playback levels with respect to tonal balance.

5. 2. The loudness normalization method of claim 1, wherein parameter smoothing is applied to the dynamic gain parameter.

6. 1. An audio processing device configured to normalize loudness based on a target loudness value, comprising: at least one processor; at least one memory storing a computer program; The at least one memory having the computer program, together with the at least one processor, configures the audio processing device to at least: determining dynamic gain parameters to be applied to audio frames of an audio signal based on short-term or long-term characteristics of the audio signal, said determining comprising determining at least one dynamic gain parameter for a first audio frame based on the short-term characteristics of the audio signal and the target loudness value, and determining at least one dynamic gain parameter for a second audio frame based on the long-term characteristics of the audio signal and the target loudness value; scaling at least a portion of the first audio frame by the at least one dynamic gain parameter for the first audio frame to modify the loudness of the first audio frame to match the target loudness upon playback; scaling the second audio frame by the at least one dynamic gain parameter for the second audio frame to modify the loudness of the second audio frame to match the target loudness upon playback; the long-term characteristics are determined differently than the short-term characteristics; Device.

7. The apparatus of claim 6 , wherein the dynamic gain parameters are identified and applied in real time.

8. 7. The apparatus of claim 6, wherein dialogue enhancements are applied that have the effect of making dialogue more salient within a particular context.

9. 7. Apparatus according to claim 6, in which loudness equalisation is applied having an effect on one or more playback levels with respect to tonal balance.

10. The apparatus of claim 6 , wherein parameter smoothing is applied to the dynamic gain parameters.

11. A storage medium carrying a software program adapted for execution on a processor to perform the method steps of any one of claims 1 to 5 when executed on a computing device.

12. A computer program product comprising executable instructions for carrying out the method of any one of claims 1 to 5.

Images