Dialogue extension in audio codecs
By performing compression and equalization on estimated speech components in audio codec systems, the subtle effects of dialogue enhancement are amplified, significantly improving speech clarity and distinction.
Patent Information
- Application Number
- JP2021560874
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-08-05
- Filing Date
- 2020-04-15
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-04-15
AI Technical Summary
Conventional audio codec systems fail to perceptibly enhance conversations due to subtle effects of dialogue enhancement parameters, and existing solutions are not readily applicable.
Perform additional processing, including compression and equalization, on estimated speech components in audio codec systems, using dialogue enhancement parameters to enhance speech components.
Enhances speech components by increasing the signal-to-noise ratio and making speech more distinct, providing noticeable improvements in audio quality.
Smart Images

Figure 0007577683000020 
Figure 0007577683000021 
Figure 0007577683000022
Abstract
Description
[Technical field]
[0001] [Related Applications] This application claims priority to the following earlier applications: US Provisional Application No. 62 / 833,855 (Reference No. D18119USP1), filed April 15, 2019, EP Application No. 19169218.5 (Reference No. D18119EP), filed April 15, 2019, and US Provisional Application No. 62 / 882,722 (Reference No. D18119USP2), filed August 5, 2019, all of which are incorporated herein by reference.
[0002] Related Fields This disclosure relates to dialogue enhancement in audio encoder-decoders (codecs). [Background technology]
[0003] Support for speech enhancement features is typically included in conventional audio coding / decoding systems.
[0004] In a dual-ended system, information for enhancing dialogue may be included in the bitstream transmitted from the encoder to the decoder. This information is typically called a set of time-varying dialogue enhancement (DE) parameters, one parameter per frequency band (per channel). The time slots along with the frequency bands (within a channel) are together called "time-frequency tiles", and the DE parameters represent the time-varying gain for each such tile. At the decoder side, the DE parameters are applied together with the user-determined dialogue gain to provide the dialogue enhancement signal.
[0005] However, the effects of speech enhancement in such systems are too subtle to be perceptible, and therefore improved processing is desirable.
[0006] In other areas, speech enhancements have been proposed that involve a combination of equalization and compression, see for example US2012 / 0209601, but such solutions are not immediately applicable to audio codec systems. Summary of the Invention
[0007] It is therefore an object of the present disclosure to provide improved dialogue enhancement in audio codec systems.
[0008] According to the present disclosure, this and other objects are achieved by performing additional processing, including compression (and optionally equalization), on the estimated speech components, thereby enabling improved speech enhancement. Embodiments of the present disclosure are based on the recognition that the speech enhancement of audio codec systems can be significantly improved. Furthermore, in dual-ended systems according to embodiments of the present invention, dialogue enhancement (DE) parameters, such as linear gains conventionally applied directly to the audio signal, are instead used to estimate the speech components, thereby enabling additional processing.
[0009] According to a first aspect of an embodiment of the present invention, additional processing of the estimated speech components is performed at the decoder side.
[0010] More specifically, a first aspect is a method for dialogue enhancement of an audio signal, comprising: receiving an encoded bitstream comprising the audio signal; obtaining a set of time-varying parameters configured to estimate speech content present in the audio signal; estimating the speech component from the audio signal; applying a compressor to only the estimated speech components to generate a processed speech component; applying a user determined gain to the processed speech components to generate enhanced speech components; combining the extended speech component with the audio signal to form a dialogue extended audio signal; The present invention relates to a method comprising the steps of:
[0011] A first aspect is a decoder for dialogue enhancement of an audio signal, the decoder obtaining a set of time-varying parameters configured to estimate dialogue components present in the audio signal, the decoder comprising: a decoder element for decoding said audio signal received in an encoded bitstream; a conversation estimation unit that estimates the conversation component from the audio signal; a compressor for compressing the estimated speech components to generate processed speech components; a gain element for applying a user determined gain to the processed speech component to provide an enhanced speech component; a combining path for combining the enhanced speech component with the audio signal to form a speech enhanced audio signal; The present invention further relates to a decoder including the
[0012] In a single-ended system, the time-varying parameters for estimating the speech component may be determined or pre-set in the decoder, however, in a preferred implementation the decoder is part of a dual-ended system, in which case the parameters may be included in the encoded bitstream (e.g. corresponding to conventionally known speech enhancement (DE) parameters).
[0013] The compressor is advantageously applied only to the estimated speech components of the audio signal. The compressor is advantageously applied before applying the user-determined gain and before combining the enhanced speech components with the audio signal. In conventional decoders, the entire audio signal, including speech and non-speech components, may be boosted during processing of the audio signal as standard. In conventional decoders, a limiter is typically applied to the boosted signal to avoid saturating the boosted signal and thus prevent clipping of the boosted signal. In a first aspect of an embodiment of the invention, the compressor has a different purpose than a conventional limiter that is typically inserted at the decoder output. The compressor according to the first aspect is used to increase the average power of only the speech components of the audio signal, while keeping the peak levels of the audio signal unchanged. The user-determined gain is applied to the processed (compressed) speech components and combined with the audio signal or, in the embodiment described below, with the non-speech components, so that the speech components stand out more clearly in the processed audio signal. Thus, according to the first aspect, the compressor increases the signal-to-noise ratio of the dialogue-enhanced audio signal between the dialogue components and non-speech components, e.g. background, of the audio signal, and therefore the compressor according to the first aspect is not used to prevent clipping of the signal.
[0014] In an embodiment, the speech component includes speech having a level that varies over time. According to a first aspect, the compressor may bring the audio level of the louder parts of the speech component closer to the audio level of the quieter parts.
[0015] In an embodiment, the compressor may apply a make-up gain to the processed (compressed) speech to increase the level, e.g., peak level, of the processed speech back to the level, e.g., peak level, of the estimated speech The application of the make-up gain results in an overall increase in the level of the speech, thus making the compressed speech more audible.
[0016] In further embodiments described below, a limiter may be used at the output of the decoder to protect the processed audio signal from clipping. If the dialogue is boosted by a simple gain but not compressed, the limiter may significantly reduce or cancel the perceived effect of the boosted dialogue. On the other hand, by compressing and boosting the dialogue, the average power of the dialogue may be increased to achieve a perceived increase in the dialogue level even after limiting. In this way, applying a compressor only to the dialogue content of the audio signal provides an output limiter with a more perceptually robust dialogue enhancement system.
[0017] It will be appreciated that when equalization is further applied to the estimated speech component prior to compression, compressing the estimated speech component refers to compressing the equalized estimated speech component.
[0018] According to a second aspect of an embodiment of the present invention, additional processing of the estimated dialogue components is performed at the encoder side of the dual-ended system, resulting in modified dialogue enhancement (DE) parameters, which are encoded and included in the bitstream.
[0019] Note that while compression is a time-variant non-linear operation, it is simply the calculation of gain values that is non-linear. The actual application of the calculated gain values is effectively a linear operation. The application of a static (time-invariant) equalizer curve is also linear. The inventors have therefore realised that additional processing of the dialogue material according to embodiments of the present invention can alternatively be achieved at the encoder side by incorporating the equalizer coefficients and compression gains (including make-up) into a set of dialogue enhancement (DE) parameters to produce a modified set of DE parameters.
[0020] More specifically, a second aspect is a method of encoding an audio signal to enable dialogue enhancement, comprising: Providing an audio signal; providing a set of time-varying speech enhancement parameters configured to estimate speech content present in the audio signal; estimating an estimated speech component by applying the speech enhancement parameters to the audio signal; applying a compressor to only the estimated speech components to generate processed speech components; dividing the processed speech components by the estimated speech components to determine a set of time-varying adjustment gains; combining said speech enhancement parameters with said adjustment gains to provide a set of modified speech enhancement parameters; encoding said audio signal and said modified dialogue extension parameters into a bitstream; The present invention relates to a method comprising:
[0021] A second aspect is an encoder for encoding an audio signal to enable dialogue enhancement, the encoder comprising: a speech estimator for estimating speech components present in an audio signal by applying a set of time-varying speech extension parameters to the audio signal; a compressor for compressing only the estimated speech components to generate processed speech components; a divider for dividing the processed speech component by the estimated speech component to determine a set of time-varying adjustment gains; a combiner for combining said speech enhancement parameters with said adjustment gains to provide a set of modified speech enhancement parameters; an encoder element for encoding the audio signal and the modified dialogue extension parameters into a bitstream; The present invention also relates to an encoder including
[0022] The advantageous effects of the compressor described with reference to the first aspect of the embodiment of the invention are also achieved by the second aspect of the various embodiments of the invention.
[0023] Both aspects (decoder and encoder) provide substantially the same technical effect.
[0024] The advantage of the second aspect (processing in the encoder) is that the decoder does not need to be modified. The compressor may attenuate parts of the signal that exceed a given threshold, e.g. parts of the signal that have peak or RMS levels higher than a given threshold. The compression ratio may be about 5:1 or up to 20:1. Make-up gain can be applied to maintain the original levels (e.g. peak or RMS levels) of the speech signal.
[0025] In a dual-ended system, the encoded bitstream may also include compression parameters for configuring the compressor. Such parameters may include, for example, thresholds, compression ratios, attack times, release times, and make-up gains.
[0026] Further processing of the estimated speech component preferably includes applying a first equalizer to the estimated speech component before applying the compressor. Such equalization may serve to further enhance the effect of the compression.
[0027] The term "equalizer" should be interpreted broadly and may include, for example, the application of a difference equation in the time domain. In most practical instances, however, an equalizer is an element that applies a frequency-dependent (complex) gain to the estimated speech signal, although in some cases a real-valued gain may be sufficient.
[0028] The equalizer may include rolling off lower frequencies (e.g., below 500 Hz) and providing a small, broad boost in a selected frequency range. See below for more detailed examples.
[0029] Combining the enhanced speech component with the audio signal may include subtracting the estimated speech component from the audio signal to form an estimated non-speech component (sometimes called music and effects (M&E)), and then adding the estimated non-speech component to the enhanced speech component.
[0030] In some embodiments, the estimated non-speech components are also equalized by applying a second equalizer before the estimated non-speech components are added to the extended speech components. Such a second equalizer may be functionally correlated with the first equalizer. For example, in frequency regions where the estimated speech is amplified, the M&E may be given a small attenuation. See the description of the embodiments for more detailed examples.
[0031] In a dual-ended system, the encoded bitstream may also include control or steering data for configuring the first equalizer and, if present, the second equalizer. For example, the decoder may be provided with a set of different equalizer presets, and control data in the bitstream may select which preset to apply. [Brief description of the drawings]
[0032] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings, in which: [Figure 1] FIG. 2 is a block diagram of a decoder according to an embodiment of the present invention; [Diagram 2] FIG. 4 is a block diagram of a decoder according to a second embodiment of the present invention; [Diagram 3] FIG. 2 is a block diagram of a portion of an encoder according to an embodiment of the present invention; [Figure 4] 4 is a decoder suitable for use with the encoder solution of FIG. [Diagram 5] FIG. 3 is a block diagram of a more detailed implementation of a decoder of the method of FIG. 2. [Figure 6] 6 is a block diagram illustrating the conversation extension component of FIG. 5 in accordance with an embodiment of the present invention. [Figure 7A] 6 is a block diagram illustrating an example of the conversation extension component of FIG. 5 according to another embodiment of the present invention. [Figure 7B] 6 is a block diagram illustrating an example of the conversation extension component of FIG. 5 according to another embodiment of the present invention. [Figure 8]6 is a block diagram illustrating the conversation extension component of FIG. 5 in accordance with yet another embodiment of the present invention. [Figure 9A] 3 is an example of an equalization function of the equalizer in FIG. 2. [Figure 9B] 3 is an example of an equalization function of the equalizer in FIG. 2. [Figure 10A] 1 illustrates a schematic example of conversation expansion according to a conventional approach. [Figure 10B] 1 illustrates a schematic diagram of an example of conversation expansion according to an embodiment of the present invention; [Figure 10C] 4 illustrates a schematic diagram of an example of conversation expansion according to another embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0033] The systems and methods disclosed below may be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, division of tasks does not necessarily correspond to division into physical units. Conversely, one physical component may have multiple functions, and one task may be performed jointly by multiple physical components. Certain or all components may be implemented as software executed by a digital signal processor or microprocessor, or as hardware, or as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term "computer storage media" includes both volatile and non-volatile, removable and non-removable media, implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or non-magnetic storage devices, or any other medium accessible by a computer which can be used to store the desired information. Additionally, as known to those skilled in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
[0034] The following description relates to various decoder and encoder embodiments in a dual-ended codec system. It is noted that embodiments of the present invention may also be implemented in a single-ended decoder. In such an embodiment, the time-varying parameters a for estimating the speech component are not received in the bitstream, but are instead determined by the decoder based on the received audio signal Y.
[0035] <Decoder implementation> The decoder 10 of FIG. 1 includes a speech estimation block 11 that receives an input audio signal Y and time-dependent dialogue enhancement (DE) parameters from a bitstream. Although not shown in FIG. 1, both the audio signal Y and the parameter set a are decoded from the encoded bitstream. The parameters a include parameters for each of a set of frequency bands (and of course for each channel carrying speech). The resolution of the time dependence is typically determined by the frame rate of the bitstream, and a particular combination of frame (m) and frequency band (k) is called a time-frequency tile. According to this terminology, the DE parameters include one or more parameters a(m,k) for each time-frequency tile. It should be noted that the DE parameters typically have a coarser frequency resolution than the audio signal, and one DE frequency band may include several frequency bins of the audio signal. The DE parameters enable the speech estimation block 11 to estimate the speech component D present in the audio signal Y according to D(m,k)=a(m,k)Y(m,k). For further details, see WO2017 / 132396, which is incorporated herein by reference.
[0036] The decoder further comprises a speech processing path which in this embodiment comprises a first equaliser 12 connected in series with a compressor 13. The output of the compressor 13 is connected to an amplifier 14 which performs a multiplication by a factor of g-1, where g is a user determined linear gain.
[0037] The user determined gain g may represent the degree of speech gain to be applied overall. For example, the user may set the gain g to a level that the user feels comfortable with and let it be at that level. If the user feels that the level of speech is too quiet, the user can increase the level by increasing the gain g. Similarly, if the user feels that the level of speech is too loud, the user can decrease the level by decreasing the gain g. However, in many practical cases the user will prefer more speech, and the gain may be typically set to a value greater than or equal to 1.
[0038] Upstream of the equalizer 12, a switch 15 is arranged which, in this embodiment, is configured to connect the estimated speech signal D to the processing path (compressor 13 and optionally equalizer 12) only if the following two conditions are met: 1) The user-selected gain factor g is greater than 1; 2) The speech extension parameter a is non-zero for the time-frequency tile, i.e., speech is present.
[0039] If neither of these conditions are met, the estimated speech component D is connected directly to the multiplier 14 without any processing. Other settings of the switches are possible, for example not requiring the second condition.
[0040] Finally, the decoder includes a summing junction 16 arranged to add the output from multiplier 14 to the input audio signal Y.
[0041] In use, when g>1, the equalizer of Figure 1 processes the estimated speech component D (by compressing and optionally equalizing), then multiplies it by g-1, and then finally adds it to the original audio signal Y. When g≦1, the encoder multiplies the estimated speech component D by g-1 (without processing) and adds it to the original audio signal Y. Note that this latter case corresponds to attenuating the speech, since the factor g-1 is less than 0. The summation point 16 is therefore a subtraction in this case.
[0042] A more elaborate embodiment is shown in Figure 2, where the decoder 20 further comprises a subtraction point 21 arranged to subtract the estimated speech D from the input audio signal Y, thereby forming an estimated "non-speech component", sometimes called M&E (music and effects). The decoder of Figure 2 further comprises a processing path with a second equalizer 22, the output of which is connected to a summing point 24. Before the second equalizer 22 there is a second switch 23 which in this embodiment is arranged to feed the M&E signal to the second equalizer only if the following two conditions are met:
[0043] 1) The user-selected gain factor g is greater than 1; 2) The speech extension parameter a(m,k) is non-zero for the time-frequency tile, i.e., speech is present.
[0044] 2, summing point 24 is connected to either the processed M&E from equalizer 22 or the unprocessed M&E directly from switch 23. The result of the summation is a dialogue-enhanced audio signal.
[0045] The equalizers 12, 22 of Figures 1 and 2 are typically configured to apply a frequency-dependent (complex) gain to the input signal (i.e., the estimated speech signal or the M&E signal). The first and second equalizers 12, 22 can be functionally related to each other in the sense that, for example, when the gain function of the first equalizer has a boost, the gain function of the second equalizer has a corresponding (but typically narrower band) cut. This is illustrated in Figures 9A, 9B for exemplary equalizer gain functions EQ1 and EQ2. The first gain function EQ1 here has a roll-off below about 400 Hz, a slight valley (i.e. attenuation) at about 3 kHz, and peaks at about 5 kHz and 10 kHz. The second gain function EQ2 has corresponding cuts at about 5 kHz and 10 kHz. It should be noted that these gain functions are merely examples, and the details of the gain functions depend on the actual application and the desired effect.
[0046] The compressor 13 of Figures 1 and 2 may be a single-ended compressor that includes parameters such as threshold, compression ratio, attack time, release time, and make-up gain. The compression parameters may be decoded from the bitstream and may vary from frame to frame. For example, the compression ratio may be 5:1, 10:1, or 20:1. The attack may be 10 ms, the release may be 250 ms, the typical threshold may be -6 dB, and the make-up gain may be 10 dB. The threshold (i.e., the lower limit of the compression to be set) may be set relative to a long-term or short-term loudness level reference. For example, it may be set relative to a speech normalization value, which may be an indication of the average speech loudness in the audio signal. The speech normalization value may possibly be adjusted for local deviations based on additional information that may be provided in the bitstream.
[0047] <Implementation as matrix multiplication> It should be noted that the block diagrams in Figures 1 and 2 are schematic representations of the decoder's functionality. A more practical implementation is typically realized as a matrix multiplication Z=H·Y, where Y is the input audio signal, H is a transfer function in the form of an input-output matrix, and Z is the speech augmented output signal.
[0048] Considering the speech enhancement in the quadrature mirror filter (QMF) domain, the input audio signal is:
number
number
[0049] Then, for a>0 and g>1, the speech extension output Z is given by:
number
number
number
[0050] Therefore, in a practical implementation, the coefficients of the static EQ curve can be stored in a look-up table, and the compression gain f only needs to be calculated from V before H is applied to Y.
[0051] In a multi-channel setup, the corresponding gain f is obtained in a similar way by calculating the gain for each channel separately and taking the minimum gain as a common gain for all channels, which is equivalent to calculating the instantaneous envelope values of each channel separately and deriving the gain from the maximum envelope value.
[0052] If additional processing of estimated conversation D is turned off (g<1), then f=q1=q2=1 and the output Z becomes:
number
[0053] The need to have additional buffers for D and / or D can be eliminated by computing the envelope V as follows:
number
[0054] <Encoder implementation> The approach described with reference to FIG. 2 can also be applied on the encoder side as shown in FIG.
[0055] The encoder of the dual-ended system includes digital processing circuitry (not shown) that computes a set of time-varying speech enhancement parameters a to be included in the bitstream, so that the decoder can estimate the speech component from the audio signal.
[0056] 3 shows part of the encoder, which includes a speech estimation block 31 (similar to speech estimation block 11 described above) for estimating the speech component D present in the audio signal Y using parameters a pre-calculated in the encoder. The encoder further includes a processing path having an equalizer 32 and a compressor 33. The processing path receives the estimated speech component D and provides the processed result to a multiplier 34 ... inverted speech component D which is provided to the multiplier 34. -1 Output x -1 It also includes an inverter 35. The output of the multiplier is connected to a second multiplier 36. The second multiplier 36 also receives the speech extension parameter a.
[0057] In use, multiplier 34 receives the processed speech component and multiplies it by 1 / D, i.e., providing a ratio r between the processed speech component and the estimated speech component. The ratio r is typically specific to one time-frequency tile. The ratio r thus represents the contribution of the processing paths 32, 33 relative to the estimated speech component for the particular time-frequency tile. For each tile, multiplier 36 multiplies the DE parameters a by the ratio r to output modified DE parameters b. The complete set of modified DE parameters b is then encoded into the bitstream together with the audio signal.
[0058] When an embodiment of the present invention is implemented on the encoder side (as shown in Figure 3), it is backward compatible with existing decoders such as the one shown in Figure 4. By multiplying the received signal Y by the modified DE parameter b, the decoder can reproduce the processed speech signal output from the compressor 33 of the encoder of Figure 3. When such a processed speech signal is scaled by g-1 and then added back to Y, it is possible to generate an improved speech enhancement signal, even in existing decoders, as shown in the decoder of Figure 4.
[0059] <Practical implementation> Figure 5 shows diagrammatically how dialogue enhancement according to a practical embodiment of the present invention can be implemented in an existing decoder topology. As shown in Figure 5, the decoder 50 typically includes a core decoder 51 for decoding the received bitstream, an optional transformer 52, e.g. a binaural transform T, a matrix transform 53 and an optional delayed reverberation processing path including a feedback delay network (FDN) block 54. The encoder further includes a dialogue enhancement (DE) block 55 providing outputs to two summing points 56, 57 (corresponding to the two summing points 21, 24 in Figure 2). Finally, the decoder 50 includes a post-processing block 58, e.g. providing a volume control, and a limiter 59.
[0060] FIG. 6 shows an example of the dialogue expansion block 55 of FIG. 5 for the specific case of stereo audio. Block 55 transforms A core The system includes a speech estimation block 61 (corresponding to block 11 in Figures 1 and 2) which estimates the speech component using f(m)·q1(k), and a processing block 62 which further provides equalization and compression of the estimated speech. If no equalization is applied to the M&E signal, the transformation in block 62 is equivalent to f(m)·q1(k), and q2(k)=1. A multiplication point 64 (corresponding to multiplier 14 in Figures 1 and 2) multiplies the processed speech by a user determined gain g.
[0061] In the illustrated embodiment, compression in block 62 is provided by a side chain 63 that calculates an appropriate compressor gain based on the estimated speech signal. It should be noted that the equalization in block 62 may also be provided upstream of a side chain branch. As a result, the input to the side chain 63 also passes through an equalizer. Another option is to apply a separate equalizer in the side chain 63. This equalizer may thus be different from the equalizer of block 62.
[0062] For stereo audio, using QMF bank notation,
number
number
number
number
[0063] The 5:1 surround case is easily derived from the stereo case. The only difference is that only the three front channels L / R / C (left / right / center) are processed by the dialogue expansion according to an embodiment of the present invention. As in the previous two-channel example, the same compressor gain is typically used for each of the three front channels. In FIG. 6, the transformation "A core " is now a 3x3 (speech) matrix, with the corresponding DE parameters as its elements, applied to only the front three channels of the 5:1 surround channel to estimate the speech signal.
[0064] 7A shows another example of a dialogue enhancement block 55' for an alternative stereo implementation, here including a binaural transform 52. For details of the binaural transform, see WO2017 / 035281 and WO2017 / 035163, which are incorporated herein by reference.
[0065] In the figure, there are three different speech estimation blocks 71, 72, 73, which represent different modes (also called "configurations") of the transformation A. cfg0 ,A cfg1 , and A cfg2 (For details, see WO2017 / 132396, which is incorporated herein by reference). It should be noted that in FIG. cfg2 A coreBlocks 62 and 63 are similar to those in FIG.
[0066] In this alternative stereo implementation, we again use the QMF bank notation and use the following formula:
number
number
number
[0067] Alternatively, the dialogue expansion 55' of Figure 7A can be applied after the stereo signal has been converted to a binaural signal (i.e. after block 52). This is disclosed in Figure 7B, where similar elements are given the same reference numbers as in Figure 7A. cfg0 Note that is used.
[0068] Subtracting the estimated speech from the stereo signal is only relevant if a binaural version of the speech is present in the bitstream. The subtraction process can be omitted at the expense of performance degradation. If the subtraction is omitted, the interpretation of the user gain g changes. In that case, a user gain equal to 0 means no speech enhancement, and a user gain equal to 1 results in a 6 dB boost. Negative values of g result in attenuation, but since the speech after speech enhancement differs from the speech of the stereo signal, a poor attenuation is expected, and therefore the speech enhancement signal at the output suffers from distortion.
[0069] In yet another embodiment, as shown in FIG. 8, a dialogue enhancement 55″ can be applied to the binaural signal prior to conversion of the dialogue enhancement stereo signal, where again similar elements are given the same reference numbers as in FIG. 7A. In this case, the configurations (cfg1, cfg2, cfg3) described above are not required and the core configuration (A core ) is needed (block 73). In effect, this variant corresponds to a cascade of 1) a stereo decoder with dialogue extensions 55", 56, 57 and 2) a binaural transform 52.
[0070] <Crossfade> In some embodiments, the decoder may be configured to switch between conventional dialogue enhancement (i.e., without dialogue compression and equalization) and dialogue enhancement according to the present invention. Such switching may be based on steering data in the bitstream, for example. For simplicity, conventional dialogue enhancement is abbreviated herein as DE, while dialogue enhancement according to the present invention is called ADE ("advanced" dialogue enhancement). Switching between DE and ADE may result in an audible jump in volume that may degrade the user experience.
[0071] To mitigate the audible effect of such discontinuities in the applied dialogue enhancement, the decoder may include a transition mechanism. The transition mechanism may be a cross-fade, which is commonly used for seamless switching. In general, a cross-fade means that the output is gradually switched from a first signal A to a second signal B over a given time. It can be expressed as follows:
number
[0072] In this case, the weighting factor can be defined by the following function: This creates a ramp from 0 to 1 when the ADE is switched on (ADE switch =1), which generates a reverse ramp from 1 to 0 when such conversation extensions are switched off (ADE switch =0).
number
[0073] The period of the ramp is determined by a time constant τ, which may be one or more decoder processing frames. In the given example, the ramp is linear, but it may be any function that smoothly transitions between 0 and 1 within the time τ. For example, it may be a logarithmic, quadratic, or cosine function.
[0074] Figures 10A and 10B show schematics of dialogue enhancement in an immersive stereo system according to a conventional approach (Figure 10A) and according to an embodiment of the present invention (Figure 10B). Figure 10B basically corresponds to Figure 7A above. However, the equalizer and compressor are now shown as application of gains calculated in the ADE gain calculation block 105.
[0075] It should be noted that an immersive stereo system is used as an example, and the cross-fade principle may also be implemented in other applications switching between DE and ADE.
[0076] In both cases, the input LoRo stereo signal is processed to provide a dialogue-enhanced immersive LaRa signal. In FIG. 10A, the matrix M DE is applied to the LoRo signal, and in Fig. 10B, the matrix M ADE is applied to the LoRo signal. To this end, a Feedback Delay Network (FDN) 100 receives the FDN feed signal and generates an FDN signal that is mixed with the extended dialogue to obtain the final headphone output LbRb.
[0077] From FIG. 10A, using the notation used earlier, we have:
number
[0078] From FIG. 10B, using the notation used earlier, we have:
number
[0079] It should be noted that the conversation extension is realized in the CQMF domain, ADE and M DEBoth I and La are 2x2 matrices varying in time slot and CQMF band, and LoRo and LaRa are 2x1 matrices (column vectors) varying in time slot and CQMF band. As stated, I2 is a 2x2 identity matrix.
[0080] The cross-faded LaRa signal, i.e., the cross-fading of the LaRa signal of FIG. 10A and the LaRa signal of FIG. 10B, is performed according to the following equation: ADE and M DE This can be achieved by directly applying the weighting function f_smooth defined above to
number
[0081] FIG. 10C illustrates this diagrammatically as a block diagram based on the diagrams of FIGS. 10A and 10B.
[0082] In Figure 10C, a weight f_smooth is applied to the output from block 104, so that the subtraction of the conversation in Figure 10B disappears as f_smooth approaches 1. Furthermore, a weight f_smooth is applied to multiplication point 106, and a weight (1-f_smooth) is applied to the gain (g-1). These two weighted gains are then added at summation point 107 before being connected to multiplication point 103. This means that with f_smooth=0, multiplication point 103 receives the same input as in Figure 10A, while with f_smooth=1 it receives the same input as in Figure 1B.
[0083] <Generalization> References throughout this specification to "one embodiment," "some embodiments," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. Thus, the appearances of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same exemplary embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.
[0084] As used herein, unless otherwise noted, the use of ordinal numbers "first," "second," "third," etc. to describe a common object is intended merely to indicate that different instances of a similar object are being referred to, and does not imply that the objects being described must be in a given sequence temporally, spatially, in ranking, or in any other way.
[0085] In the following claims and the description herein, any one of the terms: includes, has, consists, or consists is broad and means to include at least the element / feature that follows it, but not to exclude others. Thus, the term: includes, when used in the claims, should not be interpreted as being limited to the means or elements or steps listed thereafter. For example, the scope of the expression: an apparatus includes A and B should not be limited to the apparatus being composed of elements A and B only. The term: has, when used herein, is also broad and means to include at least the element / feature that follows it, but not to exclude others. Thus, has is synonymous with includes, meaning to include.
[0086] As used herein, the term "exemplary" is used in the sense of providing an example rather than denoting quality, i.e., an "exemplary embodiment" is an embodiment provided as an example, and not necessarily an embodiment of exemplary quality.
[0087] It should be understood that in the above description of exemplary embodiments of the present disclosure, various features of the disclosure are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the invention and aiding in understanding one or more of the various aspects of the present invention. However, this method of disclosure should not be interpreted as reflecting an intention that the claims require more features than are expressly recited. Rather, as the following claims reflect, aspects of the present invention lie in fewer than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present invention.
[0088] Moreover, although some embodiments described herein include some features and not other features included in other embodiments, it will be understood by one of ordinary skill in the art that combinations of features from different embodiments may form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
[0089] Additionally, some of the embodiments have been described as methods or combinations of method elements that may be implemented by a processor of a computer system or other means for performing functions. Thus, a processor with the necessary instructions for carrying out such a method or method element forms a means for carrying out the method or method element. Furthermore, the herein described elements of the device embodiments are examples of means for carrying out the functions performed by the elements for carrying out various embodiments of the present invention.
[0090] In the description provided herein, numerous specific details have been set forth. However, it will be understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in order not to obscure an understanding of the description of the invention.
[0091] Similarly, it is noted that the term "coupled" when used in the claims should not be interpreted as being limited to only direct connections. The terms "coupled" and "connected" may be used, along with derivatives. It should be understood that these terms are not intended as synonyms for each other. Thus, the scope of the expression, device A coupled to device B, should not be limited to devices or systems in which the output of device A is directly connected to the input of device B. It means that there is a path between the output of A and the input of B, which is a path that includes other devices or means. "Coupled" may mean that two or more elements are in direct physical or electrical contact, or that two or more elements are not in direct contact with each other but still cooperate or interact with each other.
[0092] Thus, while specific embodiments of the present invention have been described, those skilled in the art will appreciate that other or further modifications may be made. For example, any formulas presented above are merely representations of procedures that may be used. Functions may be added or deleted from the block diagrams, and operations may be interchanged between functional blocks. Steps may be added or deleted to methods described within the scope of embodiments of the present invention.
[0093] For example, a decoder embodying the invention may include different processing blocks than those shown in FIG.
[0094] The embodiments of the present invention relate to the following enumerated exemplary embodiments (EEEs). (EEE1) A method for dialogue enhancement of an audio signal, comprising: receiving an encoded bitstream comprising the audio signal; obtaining a set of time-varying parameters configured to estimate speech content present in the audio signal; estimating the speech component from the audio signal; applying a compressor to the estimated speech components to generate processed speech components; applying a user determined gain to the processed speech component to provide an enhanced speech component; combining the extended speech component with the audio signal to form a dialogue extended audio signal; The method includes: (EEE2) The method of EEE1, further comprising the step of applying a first equalizer to the estimated speech component before applying the compressor. (EEE3) The method described in EEE1 or EEE2, wherein the step of combining the extended speech component with the audio signal includes the steps of forming a non-speech component by subtracting the speech component from the audio signal, and adding the non-speech component to the extended speech component. (EEE4) The method of EEE3, further comprising the step of applying a second equalizer to the non-speech components before adding them to the extended speech components. (EEE5) The method of EEE4, wherein the second equalizer is functionally correlated to the first equalizer. (EEE6) The method according to any one of EEE1 to 9, wherein the set of time-varying parameters includes one parameter for each set of frequency bands. (EEE7) The method according to any one of EEE1 to 10, wherein the encoded bitstream includes the time-varying parameters. (EEE8) The method of EEE7, wherein the encoded bitstream also includes compression parameters for configuring the compressor. (EEE9) The method according to EEE7 or 8, wherein the encoded bitstream also includes steering data for configuring the first equalizer and, if present, the second equalizer. (EEE10) The method according to any one of EEE1 to 13, further comprising the step of applying a cross-fade to trigger a step of combining the extended speech component with the audio signal and, when applicable, a step of subtracting the estimated speech component from the audio signal. (EEE11) A method for encoding an audio signal to enable dialogue enhancement, comprising the steps of: Providing an audio signal; providing a set of time-varying speech enhancement parameters configured to estimate speech content present in the audio signal; estimating an estimated speech component by applying the speech enhancement parameters to the audio signal; applying a compressor to the estimated speech components to generate processed speech components; dividing the processed speech components by the estimated speech components to determine a set of time-varying adjustment gains; combining said speech enhancement parameters with said adjustment gains to provide a set of modified speech enhancement parameters; encoding said audio signal and said modified dialogue extension parameters into a bitstream; The method includes: (EEE12) The method of EEE11, further comprising the step of applying a first equalizer to the estimated speech component before applying the compressor. (EEE13) The method according to any one of EEE11 or 12, wherein the set of time-varying parameters comprises one parameter for each set of frequency bands. (EEE14) A decoder for dialogue enhancement of an audio signal, said decoder obtaining a set of time-varying parameters configured to estimate dialogue components present in said audio signal, said decoder comprising: (EEE15) A decoder element for decoding said audio signal received in an encoded bitstream; a conversation estimation unit that estimates the conversation component from the audio signal; a compressor for compressing the estimated speech components to generate processed speech components; a gain element for applying a user determined gain to the processed speech component to provide an enhanced speech component; a combining path for combining the enhanced speech component with the audio signal to form a speech enhanced audio signal; A decoder including: (EEE16) The decoder according to EEE14, further comprising a first equalizer for equalizing the estimated speech components prior to applying the compressor. (EEE17) A decoder as claimed in EEE14 or 15, wherein the combining path includes a subtractor for subtracting the speech component from the audio signal to form a non-speech component, and a summing junction for adding the non-speech component to the extended speech component. (EEE18) The decoder according to EEE16, further comprising a second equalizer arranged to equalise the non-speech components before adding them to the extended speech component. (EEE19) The decoder according to EEE17, wherein the second equalizer is functionally correlated with the first equalizer. (EEE20) The decoder according to any one of EEE14 to 18, wherein the coded bitstream includes the time-varying parameters, and the decoder element is configured to decode the time-varying parameters. (EEE21) The decoder according to EEE19, wherein the encoded bitstream includes compression parameters for configuring the compressor. (EEE22) The decoder of EEE19 or EEE20, wherein the encoded bitstream includes steering data for configuring the first equalizer and, if present, the second equalizer. (EEE23) An encoder for encoding an audio signal to enable dialogue enhancement, comprising: a speech estimator for estimating speech components present in an audio signal by applying a set of time-varying speech extension parameters to the audio signal; a compressor for compressing the estimated speech components to generate processed speech components; a divider for dividing the processed speech component by the estimated speech component to determine a set of time-varying adjustment gains; a combiner for combining said speech enhancement parameters with said adjustment gains to provide a set of modified speech enhancement parameters; an encoder element for encoding the audio signal and the modified dialogue extension parameters into a bitstream; Including the encoder. (EEE24) The encoder according to EEE22, further comprising an equalizer for equalizing the estimated speech components prior to applying the compressor. (EEE25) A computer program product comprising computer code portions which, when executed by one or more processors, cause the processors to carry out the method according to any one of EEE1 to 10. (EEE26) A non-transitory storage medium storing a computer program product according to EEE24. (EEE27) A computer program product comprising computer code portions which, when executed by one or more processors, cause the processors to carry out the method according to any one of EEE1 to 13. (EEE28) A non-transitory storage medium storing a computer program product according to EEE26.
Claims
1. A method for dialogue enhancement of an audio signal, comprising: receiving an encoded bitstream comprising the audio signal; obtaining a set of time-varying parameters configured to estimate speech content present in the audio signal; estimating the speech component from the audio signal; applying a compressor to only the estimated speech components to generate a processed speech component; applying a user determined gain to the processed speech component to provide an enhanced speech component; combining the extended speech component with the audio signal to form a dialogue extended audio signal; Including, 20. The method of claim 19, wherein the step of combining the extended speech component with the audio signal comprises the steps of forming a non-speech component by subtracting the estimated speech component from the audio signal, and adding the non-speech component to the extended speech component.
2. The method of claim 1 , wherein the compressor is configured to increase an average power of only the estimated speech components while keeping peak levels of the audio signal unchanged.
3. The method according to claim 1 or 2, wherein the method is performed by a decoder including a limiter at its output.
4. The method according to any one of claims 1 to 3, further comprising the step of applying a first equalizer to the estimated speech components before applying the compressor.
5. The method of claim 4 further comprising the step of applying a second equalizer to the non-speech components before adding the non-speech components to the extended speech components.
6. The method of claim 5 , wherein the second equalizer is functionally correlated with the first equalizer.
7. 7. The method of claim 5, wherein the first equalizer is configured to boost one or more frequency ranges of the speech components and the second equalizer is configured to cut one or more frequency ranges of the non-speech components.
8. The method of claim 7 , wherein for a given frequency range, the boost of the speech components has a boost bandwidth and the cut of the corresponding non-speech components has a cut bandwidth narrower than the boost bandwidth.
9. A method according to any one of claims 5 to 8, wherein the encoded bitstream also includes steering data for configuring the first equaliser and, if present, the second equaliser.
10. 10. The method according to claim 1, wherein the step of applying a compressor to the estimated speech component is performed if the user determined gain is greater than one and if the estimated speech component is non-zero.
11. 11. The method of claim 1, wherein applying the compressor comprises applying a make-up gain to the processed speech components to increase a level of the processed speech components back to a level of the estimated speech components.
12. A method according to any preceding claim, wherein the set of time-varying parameters comprises one parameter for each set of frequency bands.
13. The method according to any one of claims 1 to 12, wherein the encoded bitstream comprises the time-varying parameters.
14. The method of any one of claims 1 to 13, wherein the encoded bitstream includes compression parameters for configuring the compressor.
15. 1. A method for encoding an audio signal to enable dialogue enhancement, comprising the steps of: Providing an audio signal; providing a set of time-varying speech enhancement parameters configured to estimate speech content present in the audio signal; estimating an estimated speech component by applying the time-varying speech enhancement parameters to the audio signal; applying a compressor to only the estimated speech components to generate processed speech components; dividing the processed speech components by the estimated speech components to determine a set of time-varying adjustment gains; combining said time-varying speech enhancement parameters with said time-varying adjustment gains to provide a set of modified speech enhancement parameters; encoding said audio signal and said modified dialogue extension parameters into a bitstream; The method includes:
16. The method of claim 15 , wherein the compressor is configured to increase the average power of only the estimated speech components while keeping peak levels of the audio signal unchanged.
17. The method of claim 15 or 16, further comprising the step of applying an equalizer to the estimated speech content before applying the compressor.
18. A method according to any one of claims 15 to 17, wherein the set of time-varying speech enhancement parameters comprises one parameter for each set of frequency bands.
19. 1. A decoder for dialogue enhancement of an audio signal, the decoder obtaining a set of time-varying parameters configured to estimate dialogue components present in the audio signal, the decoder comprising: a decoder element for decoding said audio signal received in an encoded bitstream; a conversation estimation unit that estimates the conversation component from the audio signal; a compressor for compressing only the estimated speech components to generate processed speech components; a gain element for applying a user determined gain to the processed speech component to provide an enhanced speech component; a combining path for combining the enhanced speech component with the audio signal to form a speech enhanced audio signal; Including, The combining path includes a subtractor for subtracting the estimated speech component from the audio signal to form a non-speech component, and a summing junction for adding the non-speech component to the extended speech component.
20. 20. The decoder of claim 19, wherein the compressor is configured to increase the average power of only the estimated speech components while keeping peak levels of the audio signal unchanged.
21. 21. A decoder as claimed in claim 19 or 20, further comprising a limiter at the output of the decoder.
22. A decoder according to any one of claims 19 to 21, further comprising a first equaliser for equalising the estimated speech content before applying the compressor.
23. 23. The decoder of claim 22, further comprising a second equalizer for equalizing the non-speech components before adding them to the extended speech components.
24. 24. The decoder of claim 23, wherein the second equalizer is functionally correlated with the first equalizer.
25. 25. A decoder as claimed in claim 23 or 24, wherein the first equaliser is arranged to boost one or more frequency ranges of the speech components and the second equaliser is arranged to cut one or more frequency ranges of the non-speech components.
26. 26. The decoder of claim 25, wherein for a given frequency range, the boost of the speech component has a boost bandwidth and the cut of the corresponding non-speech component has a cut bandwidth narrower than the boost bandwidth.
27. A decoder as claimed in any one of claims 23 to 26, wherein the encoded bitstream includes steering data for configuring the first equaliser and, if present, the second equaliser.
28. A decoder according to any one of claims 19 to 27, configured to apply the compressor to the estimated speech component if the user determined gain is greater than one and if the estimated speech component is non-zero.
29. A decoder according to any one of claims 19 to 28, wherein the compressor is further configured to apply a make-up gain to the processed speech components to increase a level of the processed speech components back to a level of the estimated speech components.
30. A decoder according to any one of claims 19 to 29, wherein the encoded bitstream comprises the time-varying parameters, and the decoder element is arranged to decode the time-varying parameters.
31. A decoder according to any one of claims 19 to 30, wherein the encoded bitstream includes compression parameters for configuring the compressor.
32. 1. An encoder for encoding an audio signal to enable dialogue enhancement, comprising: a speech estimator for estimating speech components present in an audio signal by applying a set of time-varying speech extension parameters to the audio signal; a compressor for compressing only the estimated speech components to generate processed speech components; a divider for dividing the processed speech component by the estimated speech component to determine a set of time-varying adjustment gains; a combiner for combining the time-varying speech enhancement parameters with the time-varying adjustment gains to provide a set of modified speech enhancement parameters; an encoder element for encoding the audio signal and the modified dialogue extension parameters into a bitstream; Including the encoder.
33. 33. The encoder of claim 32, further comprising an equalizer for equalizing the estimated speech content prior to applying the compressor.
34. A computer program comprising computer code portions which, when executed by one or more processors, cause the processors to carry out the method according to any one of claims 1 to 18.
35. A non-transitory storage medium storing the computer program of claim 34.
Citation Information
Patent Citations
Improved ratio of speech audio to non-speech audio for elderly or hearing-impaired listeners.
JP2010518455A
Signal processing device, method and program
JP2014089420A
Signal processing device, signal processing method and program
JP2014139658A
Signal processing device and program
JP2015022236A
Binaural dialogue improvement
JP2019508947A
