Acoustic image enhancement of stereo audio

Abstract

The invention relates to a method and a system for processing stereo audio signals. The method comprises obtaining a stereo input audio signal and determining at least one acoustic image metric of the input audio signal, where the at least one acoustic image metric is indicative of a channel level difference and / or channel of the input audio signal. The method further comprises obtaining a target acoustic image metric determined from a set of reference stereo audio signals, and determining an audio processing scheme to be applied to reduce the difference metric. The method further includes processing the input audio signal with an audio processing scheme to obtain a processed audio signal.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Spanish Patent Application No. P202230692 filed on July 28, 2022, and U.S. Provisional Application No. 63 / 421,918 filed on November 2, 2022, and U.S. Provisional Application No. 63 / 491,514 filed on March 21, 2023, all of which are incorporated herein by reference in their entirety. Technical Field

[0003] The invention relates to a stereo audio processing method and a stereo audio processing system for enhancing a stereo image of an input audio signal. Background Art

[0004] Stereo audio formats are by far the most commonly used formats in music production. In addition, stereo audio formats are also largely used for other types of audio content such as voice recordings or video soundtracks. A stereo audio signal consists of a pair of sub-signals, also called channels, and usually a left channel and a right channel, intended for left and right speakers or earbuds. When stereo audio is professionally recorded, it is first recorded and mixed before the stereo audio signal is "mastered". The mastering process involves an experienced audio engineer using accurate tools and a carefully designed listening environment to adjust the stereo audio signal (e.g., channel leveling, equalization, and filtering) to produce a final version of the stereo audio signal. When adjusting a stereo audio signal, the engineer considers a variety of factors, for example, the final stereo audio signal may need to meet professional standards and be suitable for playback on many different devices used in different environments, such as on radio broadcasts, headphones, and stereo speakers.

[0005] When mastering stereo audio content, of primary importance are the properties commonly referred to as stereo image properties of a stereo audio signal. "Stereo image properties" refer to the apparent or observed spatial qualities of an audio signal when presented in a listening environment. The apparent spatiality of a stereo audio signal (also called stereo width), inter-channel phase differences, and panning are examples of stereo image properties.

[0006] During mastering, engineers process stereo recordings to achieve, for example, proper balance between channels, suitable phase relationships, and proper stereo width, taking into account the type of audio content (e.g., the type of music). For example, it is generally believed that classical music benefits from a large stereo width at low frequencies, while other types of music, such as rock or pop, benefit from a narrower stereo width.

[0007] Thus, by manually mastering professionally recorded stereo audio signals, professional audio content is generated that is suitable for playback on a wide spectrum of devices in different environments. Summary of the invention

[0008] The disadvantage of existing solutions for generating professionally mastered stereo music is that the process is labor-intensive and requires highly specialized engineers trained to operate expensive equipment. For this reason, much music content recorded by semi-professionals or amateurs, often referred to as user-generated content (UGC), is distributed without mastering, which means that the stereo image properties have not been properly adjusted. Due to the increasing popularity of, for example, amateur recording devices (e.g., in smartphones or computers), UGC has become more widespread in the past few decades, and the current consumption rate of UGC is similar to or even exceeds the consumption rate of professionally generated content PGC. Therefore, most of the stereo audio content consumed today has not undergone mastering, or has only undergone a very basic form of automatic mastering, and may have suboptimal or directly unsuitable stereo imaging properties.

[0009] For example, amateur recordings of stereo music often have a stereo width that is too wide or too narrow (with respect to the type of audio content that has been recorded (e.g., the type of music)), an inappropriate channel balance, or an inappropriate inter-channel phase relationship. The latter can, for example, produce a stereo audio signal that is perceived as "phase anomaly", a term generally used to describe the strange feeling that is brought to the listener by anomalies in the phase relationship between the channels of a stereo audio signal, especially in the low-range and mid-range frequency bands.

[0010] In view of the above, it is clear that there is a need for improved methods of processing stereo audio signals to enhance stereo image properties without requiring manual processing of the stereo audio signals by an experienced mastering engineer.

[0011] Another drawback of traditional mastering techniques is that the available tools have limited ability to restore and amplify a stereo image that is too narrow.

[0012] According to a first aspect of the present invention, there is provided an audio processing method, the method comprising: obtaining a stereo input audio signal including a specific type of audio content; and determining at least one acoustic image metric of the input audio signal from at least one frequency band of the input audio signal, the at least one acoustic image metric indicating a channel level difference and / or correlation between two channels of the stereo input audio signal in at least one frequency band. The method further comprises: obtaining a target acoustic image metric for each frequency band, the target acoustic metric being determined from a set of reference stereo audio signals, each reference audio signal including a specific type of audio content; and determining a difference metric for each frequency band based on the difference between the acoustic image metric and the target acoustic image metric. In addition, the method comprises: determining an audio processing scheme to be applied to reduce the difference metric for each frequency band and based on the difference metric; and processing each frequency band of the input audio signal using the audio processing scheme to obtain a processed audio signal.

[0013] By comparing the extracted acoustic image metric with the target acoustic image metric (i.e., based on the reference audio content), an accurate and automatic automatic stereo mastering method with little or no user interaction is obtained. The automatic stereo mastering method works well on a variety of input audio signals, no matter how large the difference in acoustic image metric is between the input audio signal and the target acoustic image metric. If the difference is small, the processing of the input audio signal is not so intense, and a portion of the processing can be bypassed, for example, in some frequency bands where the input audio signal is initially considered to be close enough to the target acoustic image metric. In addition, the automatic stereo mastering method is able to master input audio signals that are very different from the reference audio content associated with the target acoustic image metric. For example, in an extreme scenario, the input audio signal is a single-channel audio signal, and the target acoustic image metric is associated with spacious (wide) stereo audio content. Through the above-mentioned automatic stereo mastering method, an audio processing scheme will be automatically determined, thereby producing a processed audio signal that is similar to the reference audio content in terms of acoustic image properties, although the source content (input audio content) is very different from the reference audio content.

[0014] Although the automatic stereo audio method can process the input audio signal regardless of its level of similarity to the reference content, the method is also capable of automatically processing the input signal regardless of its level of quality, for example, the automatic processing method can be applied to both UGC and PGC.

[0015] In some embodiments, determining the audio processing scheme to be applied in each frequency band includes selecting a widening processing scheme or a tightening processing scheme. That is, the audio signal is automatically widened or tightened to approach the stereo width of the reference audio content.

[0016] In some embodiments, the method further includes: performing mid-side balancing on the output audio signal, the mid-side balancing including: determining a mid-side ratio of the output audio signal; determining a mid-side ratio difference between the mid-side ratio of the output audio signal and a mid-side ratio of a target acoustic image metric; and adjusting a middle audio signal and / or a side audio signal of the output audio signal to reduce the mid-side ratio difference.

[0017] Therefore, in addition to or as an alternative to the modification of the stereo width, the mid-side balance of the stereo audio signal is adjusted to be close to the reference audio signal.

[0018] According to a second aspect of the present invention, there is provided an audio processing system, the audio processing system comprising a processor connected to a memory, wherein the processor is configured to perform the method according to the first aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Aspects of the invention will be described in more detail with reference to the accompanying drawings, which show embodiments of the invention.

[0020] Figure 1 is a block diagram depicting an audio processing system according to some implementations.

[0021] Figure 2 is a flow chart describing a method for processing an audio signal according to some embodiments.

[0022] Figure 3 is a block diagram illustrating a frame and band separator according to some embodiments.

[0023] Figure 4 is a block diagram illustrating a pre-processing module according to some embodiments.

[0024] Figure 5 is a block diagram illustrating a tightening processing module according to some embodiments.

[0025] Figure 6 is a block diagram illustrating a widening processing module according to some embodiments.

[0026] Figure 7 is a block diagram illustrating a post-processing module according to some embodiments.

[0027] Figure 8 is a diagram schematically illustrating how an audio signal is divided into frames and frequency bands according to some embodiments.

[0028] Fig. 9 is a block diagram illustrating a reference signal analyzer according to some embodiments.

[0029] Fig.10is a block diagram illustrating an adjustment module according to some embodiments. DETAILED DESCRIPTION

[0030] The system and method disclosed in the present application can be implemented as software, firmware, hardware or a combination thereof. In hardware implementations, task division does not necessarily correspond to the division of physical units; on the contrary, a physical component can have multiple functions, and a task can be performed collaboratively by several physical components. Computer hardware can be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, a web device, a network router, a switch or a bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by computer hardware. Further, the present disclosure will relate to any collection of computer hardware that executes instructions individually or jointly to perform any one or more concepts discussed herein.

[0031] Some or all of the components may be implemented by one or more processors that accept a computer-readable (also referred to as machine-readable) code containing a set of instructions that, when executed by one or more processors, perform at least one method described herein. Any processor that is capable of executing (sequentially or otherwise) a set of instructions that specifies an action to be taken is included. Thus, an example is a typical processing system (e.g., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a hard drive, an SSD, RAM, and / or ROM. A bus subsystem may be included for communication between components. The software may reside in the memory subsystem and / or the processor during its execution by the computer system.

[0032] One or more processors may operate as standalone devices or may be connected to other processor(s), such as via a network. Such a network may be built on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0033] The software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, various forms of physical (non-transitory) storage media, such as EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, cassettes, tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Further, it is well known to those skilled in the art that (transitory) communication media typically embodies computer-readable instructions, data structures, program modules or other data in the form of modulated data signals or other transmission mechanisms such as carrier waves, and includes any information transfer media.

[0034] Figure 1 is a block diagram depicting an audio processing system 1 according to some embodiments. The audio processing system 1 may be referred to as an automatic stereo mastering system. Figure 2 , a flow chart for processing an audio signal is shown, and the operation of the audio processing system 1 will now be described in more detail. The audio processing system 1 and also the method for processing an audio signal can operate offline or online (e.g., substantially in real time). In offline processing, the entire audio signal file (e.g., the entire music track) is available and the entire audio signal file can be considered in any processing / analysis module or step. In online processing, only the past and current parts of the audio signal are available, wherein the optional addition of a limited look-ahead part mainly means that the current and past parts of the audio signal can be considered in any processing / analysis module or step. Offline processing is, for example, typically used when mastering an audio signal, while online processing is typically used in, for example, streaming scenarios or teleconferencing scenarios.

[0035] At step S1, the audio processing system 1 obtains an input audio signal A and optionally provides the input audio signal A to a preprocessing module 10. The preprocessing module 10 processes the input audio signal A at an optional step S2 to obtain a preprocessed audio signal B. The preprocessing module 10 is optional and will be described below in conjunction with Figure 4 In the embodiment without using the pre-processing module 10, the input audio signal A replaces the pre-processed audio signal B below.

[0036] Both the input audio signal A and the pre-processed audio signal B may be stereo audio signals. A stereo audio signal comprises a pair of stereo signals or "channels", such as a left and right L, R channel pair or a mid-side M, S channel pair.

[0037] The input audio signal A may also be a mono-channel audio signal comprising a single channel. In this embodiment, the mono-channel input audio signal A is first replicated to form a stereo input audio signal provided to the pre-processing module 10 .

[0038] The preprocessed audio signal B is provided to the frame and frequency band separation module 15, which separates the preprocessed audio signal B into a plurality of subsequent time frames and frequency bands. That is, the preprocessed audio signal B is divided into a series of consecutive time frames, which may partially overlap or not overlap at all in time. For example, each time frame may contain 40ms of 50% overlapped preprocessed audio signal B.

[0039] Then, each time frame of the preprocessed audio signal B is separated into a plurality of frequency bands. For example, each time frame is separated into two or more frequency bands (e.g., three frequency bands). In some embodiments, the time frame is separated into three frequency bands: a low frequency band including frequencies below 120 Hz, a mid frequency band including frequencies between 120 Hz and 1500 Hz, and a high frequency band including frequencies above 1500 Hz.

[0040] The time-framed and frequency-band separated preprocessed audio signal B is provided to the metric extractor 20 and the stereo width processing block 2. The metric extractor 20 is configured to extract an acoustic image metric K of the preprocessed audio signal B at step S31. The acoustic image metric K indicates at least one of a channel level difference and a correlation between channels of the preprocessed audio signal B. For example, the metric extractor determines a power ratio of a mid channel to a side channel representation of the preprocessed audio signal B and / or a cross-correlation between channels of the preprocessed audio signal B, referred to as an inter-channel cross-correlation (ICC).

[0041] By dividing the preprocessed audio signal B into frames, and dividing each frame into frequency bands, a time-frequency representation comprising a plurality of “patches” is formed, wherein each patch represents a frequency band of a frame of the preprocessed audio signal B. The metric extractor 20 may then determine an acoustic image metric for each frequency band of each frame separately, such as determining the ICC and / or the mid-side power ratio (M / S ratio) of the channel separately for each frequency band of each frame.

[0042] To determine (M / S ratio), if necessary, the metric extractor 20 may first convert the channels of the preprocessed audio signal B into mid-side channels, and then determine the power ratio between the channels. For example, assume that the preprocessed audio signal B includes left L and right R audio channels converted into a mid channel M and a side channel S. For example, the conversion from left L and right R audio channels to mid M and side S audio channels may be implemented using the following equations, for example:

[0043]

[0044] Where a is a constant, where a = 2 or

[0045] The acoustic image metric K (including, for example, ICC and / or M / S ratio) is provided to the processing selector 30. The processing selector 30 also obtains the target acoustic image metric K at step S32. T , the target acoustic image metric has a target image metric corresponding to the acoustic image metric K extracted from the preprocessed audio signal B. For example, the metric extractor 20 determines the ICC and / or M / S ratio for each frequency band and frame, and the processing selector 30 receives the target ICC and / or M / S ratio for each frequency band and time frame, a single target ICC and / or M / S ratio for all frequency bands and frames, or a mean / median target ICC and / or M / S ratio for each frequency band as the target acoustic image metric K T .

[0046] Target acoustic image metric K T has been determined from a set of reference audio signals, the set of reference audio signals comprising a particular type of audio content, such as music, speech, movie soundtracks, etc. It is also contemplated that the particular type of audio content is music of a particular genre, such as rock, pop, classical, blues, country, jazz, electronic, hip-hop, rhythm and blues (R&B), metal, or soul, or a particular type of movie soundtrack, such as action, romance, or comedy. It is also contemplated that the target acoustic image metric K T is manually determined, or the target acoustic image metric K T is determined from a set of reference audio signals and then manually modified by the user. For example, the user may select a target acoustic image metric associated with classical music, but adjust the M / S ratio or ICC of the target acoustic image metric in at least one frequency to achieve a target acoustic image metric K that is consistent with the default target acoustic image metric K. T The indicated stereo width is compared to a stereo width that is wider / narrower or more / less correlated in at least one frequency.

[0047] The determination of the acoustic image metric K and the determination of the processing scheme to be performed based thereon can be performed both offline and online. For example, in offline processing, the acoustic image metric K for each frame and each frequency band of the frame can be determined for the complete audio signal file. In online processing, the complete audio signal file will not be available, and the metric extractor 20 can then determine the continuously updated acoustic image metric based on a portion of the audio signal contained in the buffer (e.g., containing the current frame and one or more previous frames and optionally one or more future leading frames), thereby updating the determination of the audio processing scheme accordingly. At the initialization of the online processing, in some embodiments, no or only a very short portion of the audio signal is available. In such an embodiment, the processing selector 30 specifies a default processing scheme until a sufficient portion of the audio signal has been obtained to start extracting the "informative" acoustic image metric K. For example, the default setting is to use a bypass route or to perform a widening process with a predetermined decorrelation amount. Once the "informative" acoustic image metric K is available, the processing selector 30 will resume normal operation by determining the acoustic image metric difference, and determine the processing scheme to be applied based on the difference.

[0048] In some embodiments where some waiting time is acceptable, it is contemplated that the audio processing system waits for processing until a predetermined amount of leading audio signal content has been obtained (e.g., 5 seconds of content), whereby processing begins by determining an acoustic image metric K of the leading portion, and processing is then continuously updated as the contents of the buffer are replaced.

[0049] The processing selector 30 converts the target acoustic image metric K T The acoustic image metric K is compared with the target acoustic image metric K of the metric extractor 20 and an acoustic image difference is determined based on the comparison at step S4. Based on the acoustic image difference, a processing scheme to be applied in the stereo width processing block 2 is determined by the processing selector at step S5. For example, if the acoustic image metric K and the target acoustic image metric K T Including the corresponding ICC, the processing selector 30 may select the target acoustic image metric K when the ICC of the acoustic image metric K is higher than the target acoustic image metric K T When the ICC in , it is determined that the stereo width processing block 2 should apply the widening processing scheme. Therefore, the stereo image of the preprocessed audio signal B will be widened so as to become more perceptually similar to the specific type of audio content in the set of reference audio signals.

[0050] In one embodiment, the process selector 30 receives the target ICC and the target M / S ratio for each time frame band as the target acoustic image metric K T, and receives the detected ICC and the detected M / S ratio for each time frame and frequency band from the metric extractor 20 as the acoustic image metric K. Then, the processing selector 30 may determine (i) a target acoustic image metric K for each time frame and frequency band, respectively T The difference between the mean and median ICC and the mean and median M / S ratio of (ii) the acoustic image metric K and the mean and median ICC and the mean and median M / S ratio of the acoustic image metric K. Thus, for each time frame and frequency band, four values ​​of ICC are obtained from the preprocessed audio signal B: mean (b) ICC median (b) MS mean (b) MS median (b), where b is the frequency band, b = 1, 2, 3, ..., and from the target acoustic metric K T Get the corresponding four values ​​ICC mean,target (b) ICC median,target (b) MS mean,target (b) MS median,target (b).

[0051] At step S6, the processing selector 30 determines that the frequency band and time frame should be processed using the tightening processing scheme implemented by the tightening processor 40 if the following are determined:

[0052] 1.ICC mean (b) <ICC mean_target (b)

[0053] 2.ICC median (b) <ICC median,target (b) and

[0054] 3.MS median (b) <MS median,target (b).

[0055] On the other hand, if it is determined that:

[0056] 1.ICC mean (b)>ICC mean_target (b),

[0057] 2.ICC median (b)>ICC median,target (b) and

[0058] 3.MS median (b)>MS median,target (b)+slack(b)

[0059] Here, slack(b) is a value in dB that can be selected individually for each frequency band b. For example, slack(b) is about 2 dB.

[0060] If it is determined that neither the tightening nor the widening processing scheme is satisfied, the processing selector determines that the stereo width processing block 2 should be bypassed by selecting a route 50 for the time frame and frequency band. The bypass route 50 simply passes the pre-processed audio signal B forward without modifying the audio signal. For example, it may be determined that the acoustic image metric K extracted for one or more frequency bands and frames is sufficiently close to the target acoustic image metric K T , so that no stereo width processing is performed.

[0061] The following are combined Figure 5 and Figure 6 The tightening processing module 40 and the widening processing module 60 are described in more detail. In short, the stereo image processing block 2 adjusts the stereo width (by tightening or widening) to approximate the acoustic image metric to the target acoustic image metric K T The output of the stereo width processing block 2 is, for each audio frame and frequency band, a tightened audio signal C1, a bypassed audio signal C2, or a widened audio signal C3 extracted from the preprocessed audio signal B. The output of the stereo width processing block 2 is optionally provided to a mid-side weighted balancer 70. The audio signals C1, C2, C3 output by the stereo width processing block are sometimes referred to as processed audio signals.

[0062] The optional mid-side weighted equalizer 70 takes the outputs C1, C2, C3 of the stereo width processing block 2 and performs channel boosting and / or suppression at optional step S7 to form an M / S ratio equal to or at least closer to the target acoustic image metric K T Since the M / S ratios of frames and frequency bands may have changed from the preprocessed audio signal B (due to the processing performed by the stereo width processing block 2), the mid-weighted balancer 70 may be configured to at least determine the M / S ratio of each frame and frequency band of the output signals C1, C2, C3 from the stereo width processing block 2, and use the M / S ratios (referred to as detected M / S ratios) to measure the target acoustic image metric K relative to the target acoustic image metric K. T Based on the difference, the mid-side weighted balancing process of the mid-side weighted balancer 70 is controlled. Therefore, the mid-side weighted balancer 70 may include an additional metric extractor, which is identical to the metric extractor 20 and is configured to at least determine the M / S ratio of each time frame and frequency band of the output signals C1, C2, C3.

[0063] In one embodiment, the mid-weight balancer 70 determines a target acoustic image metric K for each frame and frequency band: T A difference between a target M / S ratio and the detected M / S ratio is determined, and based on the difference, one of the mid audio channel and the side audio channel is boosted or attenuated to achieve the target mid-side ratio.

[0064] Alternatively, the difference is used to determine a distance in decibels between the target M / S ratio and the detected M / S ratio. The target M / S ratio is achieved by dividing this distance in decibels in half and boosting the weaker of the middle and side audio signals by half the decibel distance and attenuating the stronger of the middle and side audio signals by half the decibel distance. For example, the detected M / S ratio may indicate that the middle channel is 10 dB stronger than the side channel, while the target M / S ratio indicates that the middle channel is 4 dB stronger than the side channel. The decibel distance is therefore 10–4=6 dB, whereby the middle audio signal is attenuated by 6 / 2=3 dB and the side audio signal is boosted by 3 dB to achieve the target M / S ratio.

[0065] To avoid attenuation / boosting too rapidly (which may be noticeable to a listener), the mean (e.g., root mean square) difference between the target M / S ratio and the detected M / S ratio may be determined across multiple frames in each frequency band and used to determine the attenuation / boost. Using the mean difference, the mid-side weighted balance will be smoothed over time, which may mitigate noticeable artifacts. In some offline implementations, the root mean square M / S ratio difference is determined in each frequency band across all frames in the audio signal file, thereby applying the same attenuation / boost to frames of the same frequency band in the audio signal file.

[0066] In some embodiments, the mid-side weighted balancer 70 obtains adjustable parameters as input, wherein the adjustable parameters include a user M / S ratio to use or a limit range that limits the amount of boost or cut applied by the mid-side weighted balancer 70 .

[0067] The mid-weighted balancer outputs a mid-weighted balanced audio signal D, which is forwarded to an optional post-processing module 80, which performs post-processing at an optional step S8 to obtain an output audio signal E. The post-processing module 80 may, for example, perform input energy matching and / or timbre preservation, as will be described below in conjunction with Figure 7It will be appreciated that the mid-side weighted balancer 70 and / or the post-processor 80 are optional and may be omitted for some embodiments. In such embodiments, the processed audio signal output by the stereo width processing block 2 is provided directly as the output audio signal E, the mid-side weighted balanced audio signal D is provided as the output audio signal E, or the processed audio signal output by the stereo width processing block 2 is provided directly to the post-processing module 80.

[0068] The output audio signal E is optionally provided to a subsequent adjustment module 95, which provides user controls to adjust the output audio signal E in an intuitive and capable manner. The adjustment module 95 is described below in conjunction with Fig.10 Described in further detail.

[0069] The preprocessor 10 is optional and may be omitted entirely in some embodiments. In these embodiments, the input audio signal A is provided directly to the frame and band separator 15, and the frame and band separated input audio signal A is provided to the stereo width processing block 2 and the metric extractor 20. Similarly, it should be understood that the mid-side weighted balancer 70 and the post-processor 80 are also optional, whereby the signal D output by the mid-side weighted balancer 70 or the signals C1, C2, C3 output by the stereo width processing block 2 may be provided as the final output signal of the audio processing system 1.

[0070] In some embodiments, the frequency band separation function of the frame and frequency band separator 15 is omitted, thereby processing the input audio signal or the pre-processed audio signal in the full frequency band. In such embodiments, the stereo width processing block 2 can switch between the three processing paths 40, 50, 60 for the full frequency band from one time frame to the next time frame, and can also select one of the three processing paths 40, 50, 60 for the full frequency band complete audio signal.

[0071] Figure 41 is a block diagram showing a preprocessor 10 according to some embodiments. The preprocessor 10 obtains an input audio signal A and provides it to a pre-analyzer 11. The pre-analyzer 11 performs a simple full-band and full-file (e.g., offline) analysis of the input audio signal A. In some embodiments, the pre-analyzer 11 determines the mean (e.g., root mean square RMS) energy or power of the full band covering all frequencies for each channel respectively. The mean energy or power of the two channels is provided to a subsequent channel rebalancer 12 together with the input audio signal A, wherein the channel rebalancer 12 boosts or attenuates one of the channels to balance the mean energy or power of the channels forming the channel-rebalanced audio signal A'. As an example, the mean power of the first channel (e.g., the left channel) is 2dB higher than that of the second channel (e.g., the right channel), whereby the channel rebalancer 12 boosts the second (right) channel by 2dB. In some embodiments, the attenuation or boost is limited to a range that can be adjusted and regulated by the user. The channel rebalancer 12 can also achieve channel balancing by remixing channels associated with higher mean power into channels associated with lower mean power. In some embodiments, the channel rebalancer 12 both boosts channels associated with lower mean power and remixes channels associated with higher mean power into channels associated with lower mean power.

[0072] The preprocessing module 10 is optional as described above, and in some embodiments, such as for online processing, the preprocessing module 10 may be omitted. Alternatively, the preprocessing module 10 is used for online processing and operates on the buffered audio content using a moving average window for channel energy levels.

[0073] exist Figure 5 , a block diagram of a tightening processing module 40 is shown that describes an application tightening processing scheme according to some embodiments. The tightening processing module 40 is Figure 1 One of three alternative processing modules of the stereo width processing block 2 is shown, the other two being the bypass routing 50 and the widening processing module 60 .

[0074] The tightening processing module 40 obtains the preprocessed audio signal B and performs phase correction using the phase correction module 41. The phase correction module 41 determines the correlation level between the channels of the preprocessed audio signal B for each frequency band and frame. Optionally, the phase correction module 41 also uses, for example, classical recursive filtering with predetermined attack and decay time constants to smooth the correlation in time to obtain a smoothed correlation level. For each frame and frequency band, the phase correction module 41 determines whether the (optionally smoothed) correlation level is lower than a predetermined threshold level. If the (smoothed) correlation level is lower than a predetermined phase correction threshold level, a predetermined channel of the preprocessed audio signal B is inverted for a specific frame and frequency band, otherwise the channel inversion is not performed. For example, the predetermined phase correction threshold level is about 0.2 or about 0.5. In some embodiments, the phase correction threshold can be adjusted by the user.

[0075] In some embodiments, the determination whether to invert a predetermined channel is performed for each frequency band of a plurality of frames, such as for all frames of the audio signal file (offline processing) or for past frames and / or all frames present in a buffer (online processing). In an example embodiment of online processing, the phase correction module 41 determines whether the (optionally smoothed) correlation level has been below a predetermined threshold for a plurality of past frames. If this is the case, the phase correction module 41 inverts a channel for future frames. To achieve this, the mean correlation level of a plurality of frames of the frequency band is determined, and if the mean is below a predetermined phase correction threshold, the predetermined channel is inverted for all frames of the plurality of frames.

[0076] Additionally, to avoid having quiet frames and loud frames have the same degree of influence on the mean correlation level of multiple frames, a weighting factor proportional to the energy level of each frame and frequency band may be applied to the corresponding correlation level. In this way, quieter frames (e.g., lower energy / power frames) will not influence the phase reversal decision as much as louder frames (e.g., higher energy / power frames).

[0077] Another alternative method for implementing quiet and loud frame weighting is to determine a percentile of the loudest frames in the plurality of frames (e.g., the loudest 30% of frames) and determine the mean correlation level for the frames in that percentile rather than the mean correlation level for all frames in the plurality of frames.

[0078] The phase-corrected audio signal B output by the phase correction module 41 is T1 The phase correction module 41 outputs the phase-corrected audio signal B. T1 Downmix to a phase-corrected single-channel downmix audio signal BT2 In some embodiments, the phase-corrected audio signal B T1 The left and right channels are included, whereby the single-channel downmixer applies the above equation 1 and determines the middle channel, which is used as the phase-corrected single-channel downmix audio signal B T2 Phase corrected single channel downmix audio signal B T2 Then it is provided to the subsequent energy recovery module 46.

[0079] The energy recovery module 46 determines a first set of energy or power levels for each frame and frequency band of the preprocessed (stereo) audio signal B by averaging the energy or power of the two channels in the preprocessed (stereo) audio signal B. Similarly, the energy recovery module 46 averages the energy or power of the two channels in the preprocessed (stereo) audio signal B for the phase-corrected single-channel downmix audio signal B determined by the previous single-channel downmixer 44. T2 The energy recovery module 46 can operate both offline (eg, processing the entire audio signal file) and online (eg, continuously processing portions of the audio signal contained in a buffer).

[0080] Optionally, the energy recovery module 46 smoothes the energy or power level of each group in time for each frequency band, for example using classical recursive filtering with predetermined attack and decay time constants, to obtain the preprocessed audio signal B and the phase-corrected single-channel downmix audio signal B, respectively. T2 The smoothed first and second sets of energy or power levels.

[0081] The energy recovery module 46 is further configured to determine, for each frequency band, a set of energy or power level differences between each element in the first and second sets of (optionally smoothed) energy or power levels. It is contemplated that the set of energy or power level differences may optionally be smoothed over time (e.g., across multiple consecutive frames) and / or frequency (e.g., across multiple consecutive frequency bands).

[0082] The energy recovery module 46 uses the set of (optionally) smoothed energy or power level differences to determine the single channel downmix audio signal B to which the phase correction is to be applied. T2 The gain of each frame and frequency band is determined to match the energy or power level of the preprocessed audio signal B. The determined gain is then applied to the phase-corrected single-channel downmix audio signal B T2 , to obtain the energy-preserved downmixed single-channel audio signal B output by the energy recovery module 46 T3 .

[0083] Optionally, to avoid excessive gain adjustment, the determined gain is first limited to a predetermined gain range before being applied to the downmixed single channel audio signal. In some embodiments, the predetermined gain range is between -10 dB and 10 dB. Within this range, the gain is maintained between -10 dB and 10 dB, while gains below -10 dB are set to -10 dB and gains above 10 dB are set to 10 dB.

[0084] Energy-saving downmixed single-channel audio signal B T3 is provided to a single channel decorrelator 48, which processes the energy-conserving downmixed single channel audio signal B T3 , to obtain the decorrelated single-channel audio signal B T4 .

[0085] In some embodiments, the single-channel decorrelator 48 includes a filter that takes into account the input single-channel audio signal B T3 In the case of T4 When B T3 With B T4 The decorrelation is maximum when the phase difference between is 90°±N*180, where N is an integer. The filter is an all-pass filter so that the phase is changed while the amplitude is substantially unaffected. Although a single all-pass filter is sufficient in some embodiments of the single-channel decorrelator 48, other embodiments utilize a single-channel decorrelator 48 in combination with at least two all-pass filters to better control the phase shift over the entire bandwidth of interest. In addition, since the all-pass filter has the risk of causing audio transient smearing, the single-channel decorrelator 48 may further include a transient detection mechanism to control the amount of decorrelation (e.g., the introduced phase shift) accordingly. For example, the control may include mixing the input signal B in a time-dependent manner T3 and the all-pass signal B T4 , where if a transient is detected, the input signal B is retained T3 , and if no transient is detected, the full-pass signal B is retained T4 This is described in more detail, for example, in “SYSTEM AND METHOD FOR REDUCING TEMPORAL ARTIFACTS FOR TRANSIENT SIGNALS IN A DECORRELATOR CIRCUIT,” filed as a PCT application and published as WO / 2015 / 017223, which is incorporated herein by reference in its entirety.

[0086] Decorrelated single channel audio signal BT4 Single channel downmix audio signal B with phase correction T2 are provided together to a single channel remixer 49. The single channel remixer 49 is configured to mix the decorrelated single channel audio signal B T4 Single channel downmix audio signal B with phase correction T2 In some embodiments, the single-channel remixer 49 mixes the audio signal B T2 , B T4 The corresponding frequency bands of the left channel C1 are combined into a full frequency band, thereby performing a remix in a single full frequency band. The tightened stereo audio signal C1 includes the left channel C1 L and right channel C1 R , so the left channel C1 L and right channel C1 R Obtained by a single channel remixer according to:

[0087] C1 L =(1-g)×B T2 +g×B T4 (Equation 3)

[0088] C1 R =(1-g)×B T2 -g×B T4 (Equation 4)

[0089] where g is the gain between 0 and 1.

[0090] The single channel remix produces a tightened version of the preprocessed audio signal B, because the tightening process is triggered when the preprocessed audio signal is associated with a stereo width that is too wide (eg, an ICC that is too low).

[0091] In some embodiments, the phase correction module 41, the single-channel down-mixer 44, and the energy recovery module 46 can operate at a finer granularity of frequency bands (such as the frequency granularity of determining the acoustic image metric K) than other parts of the audio processing system 1. To this end, the phase correction module 41 can be preceded by a fine-grained frequency band separation module that separates the pre-processed audio signal B into a plurality of fine-grained frequency bands (e.g., six, eight, or more frequency bands), whereby the energy recovery module 46 is followed by a fine-grained frequency band combiner that recombines the fine-grained frequency bands into the original set of (relatively coarser) frequency bands (e.g., a full frequency band or three frequency bands).

[0092] Figure 6 is a block diagram of a widening processing module 60 according to some embodiments. Figure 1The pre-processed audio signal B is provided to one of the three processing modules 40, 50, 60 of the stereo width processing block 2, wherein the widening processing module 60 is one of the three processing modules, which is used to determine the pre-processed audio signal B and the target acoustic image metric K in the processing selector 30. T The stereo width of the audio signal B is widened compared to a case where it is too narrow (eg, due to too high ICC).

[0093] In the widening processing module 60, the (stereo) pre-processed audio signal B is provided to a stereo decorrelator 61, which processes the pre-processed audio signal B to obtain a decorrelated stereo audio signal B W1 In most practical implementations, the preprocessed audio signal B already has some degree of decorrelation. That is, the cross-correlation is <1. However, with the target acoustic image metric K T In comparison, the preprocessed audio signal still exhibits too high a correlation, which means that a widening process is applied to approximate a specific type of audio content.

[0094] The stereo decorrelator 61 is configured to obtain a decorrelated stereo audio signal B having a lower correlation than the preprocessed audio signal B. W1 To achieve this, the stereo decorrelator 61 according to one embodiment comprises two single-channel decorrelators, one for processing each channel of the preprocessed audio signal B. Each single-channel decorrelator may be, for example, operationally equivalent to Figure 5 The single channel decorrelators 48 used in the tightening processing module 40 are shown, however the two single channel decorrelators are independent and are configured to implement decorrelation processing (eg different phase shifts) such that the resulting decorrelated single channel audio signals are decorrelated with respect to each other.

[0095] As an example, the preprocessed audio signal B has a signal labeled B L and B R Two channels (for example, B L is the left channel, and B R is the right channel), and the decorrelated stereo audio signal B W1 Including those marked as B W1,L and B W1,R Two channels (for example, B W1,L is the left channel, and B W1,R is the right channel). The stereo decorrelator 61 is configured to ensure that corr(B W1,L ,B W1,R ) <corr(B L ,B R), where corr(α,β) represents the level of cross-correlation between the independent variables α and β. By processing each channel B of the preprocessed audio signal B with a separate decorrelator L , B R , established corr(B W1,L ,B L )<1 and corr(B W1,R ,B R )<1, which in turn means that corr(B W1,L ,B W1,R ) <corr(B L ,B R ).

[0096] The decorrelated stereo audio signal B output by the stereo decorrelator 61 is W1 is provided to a metric extractor 62 which determines the decorrelated stereo audio signal B W1 The acoustic image metric K D Acoustic image metric K D At least including the decorrelated stereo audio signal B W1 The metric extractor 62 may be equivalent to the above combined Figure 1 The metric extractor 20 described herein operates in both online and offline modes.

[0097] Decorrelated stereo audio signal B W1 with the preprocessed audio signal B and with the decorrelated stereo audio signal B W1 The associated acoustic image metric K D are provided together to the stereo remixer 63. The stereo remixer 63 also obtains the target acoustic image metric K T and the acoustic image metric K of the preprocessed audio signal B. The stereo remixer 63 uses the mixing ratio g dry Performing decorrelation between the preprocessed audio signal B and the stereo audio signal B W1 Channel-by-channel mixing, where 0≤g dry ≤1, the ratio of the preprocessed audio signal B is g dry , and decorrelate the stereo audio signal B W1 The ratio is (1-g dry ). The resulting output of the stereo remixer 63 is a widened stereo audio signal C3.

[0098] Mixing ratio dryis arranged to obtain a widened stereo audio signal C3, wherein the median ICC is equal to or at least close to the target acoustic image metric K T The target median ICC (called ICC Target ). In one embodiment, g dry is determined by interpolating using the target median ICC between two values, the first value being the median ICC of the preprocessed audio signal (referred to as ICC B ), which is some non-zero value < 1, and the second value is the median ICC of the decorrelated stereo audio signal (called ICC BW1 ). That is, we should determine g that satisfies the following equation dry Values:

[0099] ICC Target =g dry ×ICC B +(1-g dry )ICC BW1 (Equation 5)

[0100] Among them, the mixing ratio g dry According to the following equation:

[0101]

[0102] g dry This determination is based on the assumption that the median ICC (ICC) of the preprocessed audio signal B is B ) and the decorrelated stereo audio signal B W1 The median ICC (ICC BW1 The median ICC value between the preprocessed audio signal B and the decorrelated stereo audio signal B W1 It is obtained by a linear combination (e.g., a mixture) of .

[0103] It is envisaged that the mixing ratio g dry Replaced by the modified mixing ratio g' dry , where the modified mixing ratio is the mixing ratio g dry Multiply by the scaling factor:

[0104] g' dry =S factor ×g dry (Equation 7)

[0105] Where, the scaling factor S factor is adjustable and determined, for example, by the user. factor <1 means that the correlation in the widened stereo audio signal C3 is smaller (giving an even wider stereo width), while the scaling factor Sfactor >1 gives more correlation in the widened stereo audio signal C3 (giving narrower stereo width).

[0106] Figure 7 A block diagram of a post-processing module 80 is depicted according to some embodiments. Figure 1 As described, the output signal D of the mid-side-focused balancer 70 is provided as an input to the post-processing module 80. The frequency band remixer 81 of the post-processing module 80 combines the resulting mid-side-focused balanced audio signals D obtained for each frequency band into a single full-band audio signal D P1 In some embodiments, in the stereo width processing block 2, a single stereo width processing scheme is selected for each frequency band of the full audio file. The selected stereo width processing scheme may be different for different frequency bands. As an example, the pre-processed audio signal is divided into three frequency bands: a low frequency band, a mid-frequency band, and a high frequency band, whereby for the low frequency band and the mid-frequency band, a stereo widening process is selected because it is consistent with the target acoustic image metric K T The stereo width in these frequency bands is too narrow by comparison, and for the high frequency band, a stereo tightening process is selected because the stereo width in this frequency band is closely related to the target acoustic image metric K T Too big in comparison.

[0107] Then, the full-band combined stereo audio signal D generated by the frequency band remixer 81 is P1 The input audio signal A of the stereo processing system 1 is supplied to the stereo timbre matcher 82. The function performed by the timbre matcher 82 is to ensure that the full-band audio signal D P1 The spectral envelope of the input audio signal A is the same as or at least similar to the spectral envelope of the input audio signal A. The processing performed by the stereo timbre matcher 82 is similar to that performed by the combined Figure 5 The processing performed by the energy recovery module 46 is described, the main difference being that the stereo timbre matcher 82 operates on a stereo audio signal, whereas the energy recovery module operates on a single channel audio signal. As for the energy recovery module, the timbre matcher 82 can operate in an online mode and in an offline mode, wherein in the online mode the contents present in the buffer are taken into account, and in the offline mode the entire audio signal can be taken into account.

[0108] The stereo tone matcher 82 obtains the full-band audio signal D from the frequency band remixer 81. P1 and the input audio signal A of the stereo processing system 1. The stereo timbre matcher 82 determines the energy level of each channel and frequency band for each audio signal. That is, for each channel, frequency band and time frame, the timbre matcher 82 determines the energy level of the input audio signal A, and similarly determines the energy level of the full-band audio signal D. P1Optionally, the stereo timbre matcher 82 smoothes the energy levels in time (e.g., by convolving with a smoothing kernel across frames). The stereo timbre matcher 82 determines, for each audio signal and frequency band, an average energy level of at least two channels in each audio signal based on the determined (optionally smoothed) energy levels.

[0109] Thus, an average energy level is obtained for each audio signal, frequency band and time frame. The stereo timbre matcher 82 determines the energy level difference (e.g., expressed in dB) between the input audio signal A and the full-band audio signal DP1. The energy level difference for each frequency band and time frame is used as a timbre gain, and optionally, the determined timbre gain is smoothed in time and / or frequency (e.g., using a smoothing kernel extending in the time and / or frequency dimensions).

[0110] Optionally, the (smoothed) timbre gain is also limited to a timbre gain range to avoid over-suppression or over-boosting of the audio signal which may cause noticeable acoustic artifacts. The timbre gain range is, for example, from -10 dB to 10 dB or from -6 dB to 6 dB and can be adjusted by the user.

[0111] The (optionally smoothed and / or limited) timbre gain is then applied to the full-band audio signal D P1 The corresponding time frame and frequency band of the output audio signal D is formed after the frame and frequency band division. P2 The output audio signal D after frame and frequency band division P2 is provided to an output overlap and add buffer 83 which combines the time frames and frequency bands into a single full-band audio signal which is provided as the output audio signal E of the audio processing system.

[0112] As for the combination Figure 5 With the discussed phase mixing module 41, the mono down-mixer 44 and the energy recovery module 46, the stereo timbre matcher 82 may also benefit from operating at a finer granularity frequency band than, for example, the frequency band used by the stereo width processing block 2. In such an embodiment, the stereo timbre matcher 82 may be configured to first perform a frequency band separation process to separate the frequency bands of the frequency band remixer 81 into a plurality of fine granularity frequency bands, then perform the above-mentioned processes in these fine granularity frequency bands, and finally recombine the frequency bands into the frequency bands used by the frequency band remixer 81.

[0113] The division of a full-band audio signal into one or more frequency bands or the division of an audio signal that has been band-divided into frequency bands with finer granularity can be achieved in different ways. For example, complementary shelf filters, bandpass filters, filters in the frequency domain (e.g., FFT filters), or QMF filter banks can be used. It is desirable that the filters are designed so that they ensure good reconstruction in the region where adjacent frequency bands overlap. For example, in the FFT domain, overlapping filters (e.g., bell-shaped) filters that sum to one in the overlapping region can be used. As another example, a triangular filter can be used in the FFT domain with 50% overlap, where the subsequent triangular filter begins to linearly ramp up at the center of the current triangular filter, and the current filter linearly ramps down to zero where the subsequent frequency band has its peak.

[0114] In the above, the frequency band remixer 81 combines the frequency bands to allow full band processing in the stereo timbre matcher 82. In some embodiments, Figure 1 The mid-side-focused balancer 70 also operates on the full-band representation, which means that the band remixer 81 can also be placed upstream of the mid-side-focused balancer 70, allowing the mid-side-focused balancer 70 and the stereo tone matcher 82 of the post-processing module 80 to operate on the full-band representation.

[0115] refer to Figure 8 , shows a diagram schematically illustrating how an audio signal is divided into a plurality of frequency bands. Time t is indicated along the horizontal axis and frequency F is indicated along the vertical axis. Boxes BL1, BL2, BM1, BM2, BH1, BH2 indicate respective frequency bands of a channel of an audio signal in a particular time frame. The boxes to the right of the boxes BL1, BL2, BM1, BM2, BH1, BH2 indicate the next time frame, and the boxes to the right of these boxes indicate the second next time frame, and so on. Figure 1 The different components of the audio processing system 1 shown in can operate at different granularity levels (e.g., resolution levels) in time and / or frequency. For example, in some embodiments, the preprocessor 10 will operate on a single full-band representation of the input audio signal (e.g., all frequency bands BL1, BL2, BM1, BM2, BH1, BH2 are combined into a single frequency band), while the stereo width processing block operates on an audio signal divided into two or more (e.g., three) frequency bands. In some embodiments, the stereo width processing block 2 operates using a high frequency band BH (including frequencies above 1500 Hz), a mid-frequency band BM (including frequencies between 120 Hz and 1500 Hz), and a low frequency band containing frequencies below 120 Hz, but this selection of frequency bands is merely exemplary.

[0116] Additionally, some processing modules may benefit from operating with finer frequency granularity (e.g., higher frequency resolution and more frequency bands). To this end, the high, mid, and low frequency bands may be subdivided into smaller frequency bands, such as Figure 8 As shown, the high frequency band BH includes two sub-bands BH1 and BH2, which cover a narrower frequency range than the full high frequency band BH. A processing module that can benefit from operating on a finer granularity frequency band is a stereo tone matcher 82 (combined with Figure 6 described), phase correction module 41, single channel down mixer 44 and energy recovery module 46 (combined Figure 5 For example, these modules may operate using six or eight or more frequency bands, whereas the stereo width processing selector 30 that determines whether the frequency band is to be widened or narrowed operates using three frequency bands.

[0117] Switching from one time and / or frequency resolution to another may be accomplished by any of a number of methods known in the art. For example, a full-band audio signal may be reconstructed from a first time and / or frequency resolution, whereby the full-band audio signal is used to construct an audio signal representation having a different second time and / or frequency resolution.

[0118] Fig. 9 A block diagram illustrating a reference signal analyzer 90 is shown, which is configured to determine a set of target acoustic image metrics K T The reference signal analyzer comprises an acoustic image metric extractor 92, which is configured to extract an acoustic image metric from the stereo audio signal. The acoustic image metric extractor may be combined with the above Figure 1 The target acoustic metric database 93 is the same as the metric extractor 20 described above. A reference stereo audio signal is provided to an acoustic image metric extractor 92 from a database 91 containing a specific type of reference audio content. The acoustic image extractor 92 then determines the acoustic image metrics from the reference audio content and stores them in a target acoustic metric database 93. For example, the acoustic image metric extractor 92 divides each audio channel of the reference stereo audio signal from the reference audio content into a plurality of frequency bands and time frames, and determines one or more acoustic image metrics of the reference stereo audio signal for each time frame and frequency band. For example, the acoustic metric extractor 92 determines the ICC and M / S ratio for each time frame and frequency band of the reference stereo audio signal, and then calculates the mean and median ICC and the mean and median M / S ratio of the reference stereo audio signal.

[0119] In some embodiments, the reference audio content includes at least two reference stereo audio signals (e.g., two different music soundtracks of the same genre or two different movie soundtracks), and the acoustic image metric extractor 93 determines the mean and median ICC and median MS ratio (in dB) of all stereo audio signals in the at least two stereo audio signals. In this way, a type-specific target acoustic image metric K may be obtained. T , which indicates a specific type of average acoustic image metric across multiple reference stereo audio signals. This type-specific acoustic image metric can be used as the target acoustic image metric M T Provide to Figure 1 The audio processing system 1 shown in .

[0120] The audio content of a specific type may be, for example, one of music, speech, or a movie soundtrack. The audio content of a specific type may be, for example, music of a specific genre, such as rock, pop, classical, blues, country, jazz, electronic, hip-hop, rhythm and blues (R&B), metal, or soul. It is also contemplated that the target metric database 93 may store target acoustic image metrics associated with different specific audio content types simultaneously, and the most appropriate acoustic image metric may be selected automatically or based on the user's input (e.g., indicating the acoustic image attributes of metal music that one wants to imitate) by the audio processing system 1.

[0121] As an example of automatic target acoustic image metric selection, Figure 1 The audio processing system 1 in may include an audio type classifier. The audio type classifier may, for example, be configured to perform a spectral analysis and / or analysis of metadata to predict the type of audio content included in the audio signal to be processed. For example, the classifier predicts that the input audio signal includes classical music. The audio processing system 1 may then automatically select a target acoustic image metric corresponding to this type of audio content. According to the above example, the audio processing system will then select a target acoustic image metric K associated with classical music. T It is also contemplated that the classifier may be implemented using a neural network trained to predict the type of audio content comprised in the input audio signal A.

[0122] Fig.10 A block diagram is shown describing an adjustment module 95, which can be used to fine-tune the Figure 1 The output audio signal E is obtained by the audio processing system 1 shown in FIG. The output audio signal E has been processed so as to have acoustic image properties similar to or identical to the acoustic image properties of a specific type of reference audio content. Therefore, the output audio signal E can be used directly (e.g., transmitted, stored in a storage medium, or played back).

[0123] In some embodiments, the user may wish to further fine-tune the output audio signal E, and the fine-tuning module in FIG11 provides this type of fine-tuning. The output audio signal E is provided to the first mixer 96 of the adjustment module 95, which combines the output audio signal E with the phase-corrected energy-preserving single-channel downmix audio signal B from the tightening process. T3 and the decorrelated stereo audio signal B from the widening processing module W1 As the decorrelated stereo audio signal B from the widening processing module W1 As an alternative, a completely decorrelated stereo audio signal may be obtained from the single-channel remixer in the tightening processing module, and the completely decorrelated stereo audio signal may be used to replace the decorrelated stereo audio signal B. W1 This can be achieved by setting g=1 in the above equations 3 and 4, thereby obtaining two audio signals C1L and C1R that are equal but have different signs.

[0124] The user can set a width control parameter, which indicates whether the output audio signal E should be widened or tightened. If the output audio signal is to be tightened, more phase correction energy is saved in the single-channel downmix audio signal B. T3 into the mix, and if the output audio signal is to be widened, more of at least one decorrelated stereo audio signal B W1 , C1 are introduced into the mix. The remixing can be done in the full frequency band or in multiple sub-bands. For example, the user can specify whether the width adjustment mixing of the mixer 96 is done in the full frequency band or in multiple frequency bands independently. If the latter example is selected, the user can specify for each frequency individually whether and to what extent the frequency band should be widened or tightened. The resulting audio signal output by the mixer 96 is called the enhanced output audio signal E. E1 .

[0125] Enhanced output audio signal E E1 is provided to the second mixer 97, which combines the enhanced output audio signal E E1 The second mixer 97 is mixed with the input audio signal A to obtain an adjusted output audio signal F. For example, remixing the input audio signal A can ensure that some desired acoustic properties that are lost or distorted in the processing are reintroduced into the adjusted output audio signal F. The mixing ratio of the second mixer is controlled by a wet / dry control parameter that controls the wetness or dryness of the adjusted output audio signal F. If the audio signal is mainly or entirely composed of processed audio content, it is called "dry", and if the audio signal is mainly or entirely composed of unprocessed original audio content, it is called "wet". Therefore, by controlling the wet / dry control parameter that adjusts the mixing ratio of the second mixer 97, the wetness / dryness of the adjusted output audio signal F can be adjusted.

[0126] As with the first mixer 96, it is contemplated that the second mixer may operate in full frequency band or independently for multiple frequency bands, wherein in the latter case a user is able to specify separate wet / dry control parameters for each frequency band in the latter case.

[0127] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout the disclosed discussion, terms such as "processing," "computing," "calculating," "determining," "analyzing," etc. are utilized to refer to the actions and / or processes of computer hardware or computing systems or similar electronic computing devices to manipulate and / or transform data represented as physical (e.g., electronic) quantities into other data similarly represented as physical quantities.

[0128] It should be understood that in the above description of the exemplary embodiments of the present invention, various features are sometimes grouped together in a single embodiment, figure or description thereof to simplify the present disclosure and help understand one or more aspects of the various creative aspects. However, the method of the present disclosure should not be interpreted as reflecting the intention that the claimed invention requires more features than the features explicitly stated in each claim. On the contrary, as reflected in the following claims, the creative aspects are less than all the features of the single aforementioned disclosed embodiment. Therefore, the claims following the specific embodiments are hereby expressly incorporated into the specific embodiments, wherein each claim is independently a separate embodiment of the present invention. In addition, although some embodiments described herein include some features included in other embodiments without including other features included in other embodiments, as will be understood by those skilled in the art, the combination of features of different embodiments is intended to be within the scope of the present invention and form different embodiments. For example, in the following claims, any of the embodiments claimed for protection can be used in any combination.

[0129] In addition, some embodiments herein are described as a combination of methods or method elements that can be implemented by a processor of a computer system or by other components that perform functions. Therefore, a processor with an instruction for performing this method or method element forms a component for performing a method or method element. It should be noted that when the method includes multiple elements (e.g., several steps), unless otherwise stated, any order of these elements is not implied. In addition, the elements of the device embodiments described herein are examples of components that perform the functions performed by the elements in order to perform embodiments of the present invention. In the description provided herein, many specific details are set forth. However, it should be understood that embodiments of the present invention can be practiced without these specific details. In other examples, well-known methods, structures, and techniques are not shown in detail to avoid blurring the understanding of this specification.

[0130] Those skilled in the art will recognize that the present invention is by no means limited to the above-described embodiments. On the contrary, multiple modifications and variations within the scope of the appended claims are possible. For example, the division of the audio signal into different frequency bands as described above can be accomplished in many different ways, and it is understood by the skilled person that fewer or more frequency bands can be used with the same processing techniques. It should also be noted that the audio processing system is applicable to many different specific types of audio content, such as speech or music, and the system can be configured to process audio signals both offline (allowing, for example, analysis of complete audio files) and online (substantially in real time with a limited amount of look-ahead).

Claims

1. An audio processing method, comprising: obtaining a stereo input audio signal including audio content of a particular type; determining at least one acoustic image metric of the input audio signal from at least one frequency band of the input audio signal, the at least one acoustic image metric being indicative of a channel level difference and / or a correlation between two channels of the input audio signal in the at least one frequency band; obtaining a target acoustic image metric for each frequency band, the target acoustic metric being determined from a set of reference stereo audio signals, each reference audio signal comprising the particular type of audio content; determining, for each frequency band, a difference metric based on a difference between the acoustic image metric and the target acoustic image metric; determining, for each frequency band and based on the difference metric, an audio processing scheme to be applied to reduce the difference metric; as well as Each frequency band of the input audio signal is processed using the audio processing scheme to obtain a processed audio signal.

2. A method according to any one of the preceding claims, wherein: The acoustic image metric and the target acoustic image metric each include at least one of the following: mid-channel to side-channel power ratio, and Inter-channel cross-correlation ICC measure.

3. A method according to any one of the preceding claims, wherein: Determining the audio processing scheme to be applied in each frequency band includes selecting a widening processing scheme or a tightening processing scheme.

4. A method according to any one of the preceding claims, wherein: The acoustic image metric and the target acoustic image metric include a mid-channel to side-channel power ratio and an ICC measure, wherein if the mid channel to side channel power ratio and the ICC measure of the acoustic image metric in the at least one frequency band are respectively lower than the target acoustic image metric, then a tightened audio processing scheme is applied in the at least one frequency band, applying a widening audio processing scheme in the at least one frequency band if the mid channel to side channel power ratio of the acoustic image metric and the ICC measure are respectively higher compared to the target acoustic image metric, Otherwise, the input audio signal is used as the processed audio signal in the at least one frequency band.

5. The method according to claim 3 or claim 4, wherein: The tightened audio processing solution includes: generating a single channel downmix audio signal for the at least one frequency band based on the input audio signal; processing the single-channel downmix audio signal using a decorrelator to obtain a decorrelated single-channel downmix audio signal; forming a first channel of the processed audio signal based on a weighted sum of the single channel downmix audio signal and the decorrelated single channel downmix audio signal; and A second channel of the processed audio signal is formed based on a weighted difference of the single channel downmix audio signal and the decorrelated single channel downmix audio signal.

6. The method according to claim 5, further comprising performing phase correction on the input audio signal, the phase correction comprising: determining an ICC measure for the two channels of the input audio signal for each of the at least one frequency band; as well as If the ICC measure is below a predetermined threshold, one of the two channels of the input audio signal is inverted for the at least one frequency band.

7. The method according to claim 5 or claim 6, further comprising energy matching the downmix audio signal with the input audio signal, the energy matching comprising: determining a spectral energy level in each of the at least one frequency band of the input audio signal; determining a spectral energy level in each of the at least one frequency band of the single channel downmix audio signal; determining, for each at least one frequency band, a spectral energy level difference between the input audio signal and the single-channel downmix audio signal; as well as An energy matching gain is applied to the single channel downmix audio signal in each frequency band, the energy matching gain being based on the spectral energy level difference so as to reduce the spectral energy level difference when the gain is applied to the single channel downmix audio signal.

8. The method according to claim 7, wherein: The input audio signal and the downmix audio signal comprise a set of consecutive frames, the method further comprising: The spectral energy levels are smoothed, the spectral energy level differences are smoothed, and / or the energy matching gains are smoothed over a plurality of frames.

9. The method according to any one of claims 4 to 8, wherein: The widening audio processing scheme includes separately processing the at least one frequency band of each channel of the input audio signal using a decorrelator to form a decorrelated stereo audio signal; as well as At least one frequency band of the decorrelated stereo audio signal is mixed with the input audio signal at a mixing ratio to obtain the processed audio signal.

10. The method according to claim 9, further comprising determining an ICC measure for the at least one frequency band of the channels of the decorrelated stereo audio signal; and The mixing ratio is determined by interpolating between the ICC measure of the input audio signal and the ICC measure of the decorrelated audio using an ICC measure of the target acoustic scene metric.

11. The method according to claim 10, wherein: The mixing ratio is based on a ratio between the first difference and the second difference; wherein the first difference is a difference between an ICC measure of the target acoustic image metric and the ICC measure of the decorrelated audio signal, and Therein, the second difference is a difference between the ICC metric of the acoustic image metric of the input audio signal and the ICC metric of the decorrelated audio signal.

12. The method of any one of the preceding claims, further comprising performing a mid-side weighted balancing on the processed audio signal, the mid-side weighted balancing comprising: determining a mid-to-side ratio of the processed audio signal; determining a mid-to-side ratio difference between the mid-to-side ratio of the processed audio signal and the mid-to-side ratio of the target acoustic image metric; as well as A mid audio signal and / or a side audio signal of the processed audio signal is adjusted to reduce the mid-to-side ratio difference.

13. The method according to any one of the preceding claims, further comprising performing timbre adjustment on the processed audio signal, the timbre adjustment comprising: determining a spectral energy level of at least one frequency band of the processed audio signal; determining a spectral energy level of at least one frequency band of the input audio signal; determining, for each of the at least one frequency band, a timbre difference between the spectral energy level of the processed audio signal and the spectral energy level of the input audio signal for the at least one frequency band; as well as A timbre gain is applied to at least one processed audio signal based on the timbre difference, the timbre gain causing the timbre difference to be reduced.

14. The method according to any one of the preceding claims, further comprising pre-processing the input audio signal, wherein: The pre-processing comprises: determining an overall signal level across all frequency bands for each channel in the input audio signal; determining a pre-processing difference based on the difference between the total signal levels for each channel; and A preprocessing gain is applied to at least one of the channels of the input audio signal based on the preprocessing difference to reduce the preprocessing difference.

15. The method according to claim 14, wherein: The input audio signal comprises a set of consecutive frames, and wherein determining the total signal level of each channel comprises: The mean, median or nth root of the mean of the total signal level is determined for the frame for each channel.

16. A method according to any one of the preceding claims, wherein: The target acoustic image metric has been determined as an average acoustic image metric of the set of reference audio signals comprising the specific type of audio content.

17. A method according to any one of the preceding claims, wherein: The specific type of audio content is music, preferably a specific music genre.

18. A method according to any one of the preceding claims, wherein: The at least one frequency band is at least two frequency bands.

19. The method according to claim 18, further comprising: The at least two frequency bands of the processed audio signal are combined into a full-band processed audio signal.

20. The method according to any one of the preceding claims, further comprising: Get the user-specified tuning parameters; as well as The target acoustic image metric is adjusted based on the user-specified adjustment parameter.

21. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to any one of claims 1 to 19.

22. A computer-readable storage medium storing the computer program product according to claim 21.

23. An audio processing system comprising a processor connected to a memory, wherein: The processor is configured to perform the method according to any one of claims 1 to 20.

Images