Avoid gain scaling apparatus and method for spatial, immersive, single-channel or multi-channel reproduction layouts

Abstract

An apparatus for generating one or more audio output channels from one or more audio signals is provided. The apparatus comprises a gain determiner (110) configured to generate or receive gain information indicating, for each audio signal of at least one of the one or more audio signals, how to attenuate or amplify the audio signal according to a position and a reference position associated with the audio signal. Further, the apparatus comprises a signal generator (120) configured to generate one or more audio output channels from at least one of the one or more audio signals by attenuating or amplifying the at least one audio signal according to the gain information.

Description

[0001] manual

[0002] The present invention relates to audio ducking, and more particularly to a ducking gain scaling apparatus and method for spatial, immersive, single-channel or multi-channel reproduction layouts.

[0003] In many systems, object mixing can be controlled by defining ducking sequences that apply time-varying (fading) gains on specific audio objects. For example, when foreground speech is active, attenuate the background. By default, all channels of a multichannel background signal are attenuated equally. This causes a perceived loss of sound pressure (holes) in spatial areas where the foreground signal is not present (e.g. behind the listener). This problem is exacerbated in immersive speaker layouts.

[0004] Ducking is an audio production technique for mixing two or more audio component signals using a signal-dependent, time-varying gain. In the example of two component signals, such as a dialogue or speech signal and a background signal, when the dialogue signal is active, the background signal is attenuated by a certain amount. This allows the overall level of the mixed signal to remain approximately constant, rather than mixing the dialogue signal over a constant-level background signal, which results in a higher overall level of the mixed signal. Additionally, by attenuating the background signal, such as by applying a time-varying gain, the dialogue becomes more intelligible and the listening effort is reduced. Another example of ducking is when a DJ or radio host is speaking over music, and the level of the music is reduced ("ducking") during the speech.

[0005] In channel-based audio transmission, the component signals are mixed during the production process, and only this complete mix is ​​transmitted.

[0006] In object-based audio (OBA), component audio signals or their submixes and mixing instructions are transmitted separately, and the final mixing is performed in the decoding and rendering devices, which may take into account additional end-user interaction operations.

[0007] One of these mixing instructions is a time-varying ducking gain. In other words, ducking can occur during production (a destructive process), or the ducking instructions can be transmitted along with the component audio and ducking can be performed during the rendering phase (a non-destructive process). These alternatives have slightly different boundary conditions, but all aim to achieve the same result for the end user's listening experience.

[0008] In the following, the terms "dialog", "foreground" or "main signal" may be interpreted as corresponding to a signal controlling ducking, for example.

[0009] For example, the terms "background" or "auxiliary signal" may be interpreted as corresponding to the signal to which ducking is applied. i (t) represents the background signal, b represents the background signal i (t) denotes, where i∈[1,N] is the channel index of the total N channels, and t is the time index. If the two signals have different numbers of channels, the missing channels may be filled with zeros, for example, or a format conversion process may be employed, for example.

[0010] Some existing methods of performing ducking rely on creating a single time-varying gain that is applied to all channels of the background signal. Considering OBA transmission, this can easily be done more efficiently because only a single gain signal is transmitted instead of setting up dedicated gain signals for each audio signal. In the most advanced ducking applications or audio mixing tools, the background signal is attenuated according to the single ducking gain.

[0011] A speaker layout that achieves immersive playback can, for example, be considered immersive with more than 2 speaker channels (e.g., any channels above stereo). The main purpose of an immersive playback layout is to place speakers in multiple physical locations in order to more accurately reproduce the 3D direction of the sound source. Examples of common immersive layouts are 5.1 (front left, front right, center, surround left, surround right, LFE), or, for example, a 22.2 layout. Although an increase in the number of speakers provides greater artistic freedom in the design of the audio scene, dialogue content is typically placed at the front of the audio scene and reproduced primarily (if not exclusively) by the center speaker, while the other channels provide more support for immersive background signals.

[0012] The current solution to ducking in immersive layouts is to design a universal ducking gain that is aesthetically acceptable when applied equally to all channels. This is usually a compromise between the ducking depth needed to make the foreground signal more prominent and the need to duck unnecessary channels. For example, additional per-channel scaling values ​​can be set manually, but such manual settings are laborious to design and do not guarantee optimal results.

[0013] The object of the invention is to provide an improved audio ducking concept. The object of the invention is solved by an apparatus according to claim 1, an apparatus according to claim 7, an apparatus according to claim 8, a method according to claim 31, a method according to claim 32, a method according to claim 33 and a computer program according to claim 34.

[0014] A device for generating one or more audio output channels from one or more audio signals is provided. The device includes a gain determiner configured to generate or receive gain information, wherein the gain information indicates, for each audio signal of at least one audio signal among the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position. In addition, the device includes a signal generator configured to generate the one or more audio output channels from the one or more audio signals by attenuating or amplifying the at least one audio signal according to the gain information.

[0015] In addition, according to an embodiment, a device for encoding one or more audio signals in a bitstream is provided. The device includes a gain determiner configured to generate gain information, wherein the gain information indicates, for each audio signal of at least one audio signal in the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position. In addition, the device includes a bitstream generator configured to encode the one or more audio signals in a bitstream, wherein the bitstream generator is configured to generate a bitstream so that the bitstream includes the gain information.

[0016] In addition, according to another embodiment, a device for obtaining gain information is provided. The device includes a user interface, the user interface includes an input device that enables a user to input input information, wherein the input information includes gain information, or wherein the device is configured to obtain the gain information from the input information, wherein the gain information indicates, for each audio signal of at least one audio signal among one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position.

[0017] In addition, according to a further embodiment, a method for generating one or more audio output channels from one or more audio signals is provided. The method comprises:

[0018] - generating or receiving gain information, wherein the gain information indicates, for each of at least one of the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position. And:

[0019] - generating one or more audio output channels from the one or more audio signals by attenuating or amplifying at least one audio signal according to the gain information.

[0020] In addition, according to another embodiment, a method for encoding one or more audio signals in a bit stream is provided. The method includes:

[0021] - generating gain information, wherein the gain information indicates, for each of at least one of the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position. And:

[0022] - encoding one or more audio signals in a bitstream, wherein the bitstream is generated such that the bitstream comprises the gain information.

[0023] In addition, according to a further embodiment, a method for obtaining gain information is provided. The method includes:

[0024] - receiving input information via an input device of a user interface, wherein the input information comprises gain information, or wherein the method comprises deriving the gain information from the input information, wherein the gain information is for each of at least one audio signal of the one or more audio signals as a function of a position associated with the audio signal and a reference position,

[0025] Indicates how to attenuate or how to amplify the audio signal.

[0026] Furthermore, computer programs are provided, wherein each computer program is configured to implement one of the above methods when executed on a computer or a signal processor.

[0027] According to an embodiment, on the decoder side, the decoder may, for example, apply scaled ducking and may generate an audio output for each channel, for example according to a gain sequence or, for example, according to layout information.

[0028] In another embodiment, at the encoder side, the encoder may, for example, generate scaled coefficients and may, for example, encode them in a bitstream.

[0029] However, embodiments are not limited to application only to automatic ducking. Instead, some embodiments may also be used to support improving the auditory experience produced by manual mixing.

[0030] According to further embodiments, manual audio manipulation may be supported, for example. For example, an audio engineer may be provided with a means to manipulate, for example, a gain controller of the audio content. The input audio content may be scaled according to the gain and output speaker layout, and, for example, a scaled output audio content may be generated.

[0031] Some embodiments provide a solution for scaling the ducking depth per channel, depending on the spatial location of the channel and the single-channel or multi-channel layout to which it belongs.

[0032] According to an embodiment, a concept is provided for designing a per-channel correction value of the ducking gain so that the spatial holes are reduced, but the overall perceived ducking depth is still equal to before the correction and is comparable in all speaker layouts.

[0033] In an embodiment, this may be accomplished, for example, in two steps: reducing the ducking depth (per-channel ducking scale) as the channel is farther from the foreground signal location, and adjusting the scale so that the energy of the ducked signal is approximately the same when the same ducking is applied to all channels and when using per-channel ducking values.

[0034] According to an embodiment, the provided modification gain may be applied to MPEG-H transmission via the DRC modification parameters that are currently being used, for example.

[0035] In an embodiment, implementation may include, for example, pre-calculating correction gains for supported speaker layouts and using them as pre-defined constants during content authoring.

[0036] According to an embodiment, the provided concept may include, for example, a production-side tool suitable for offline and real-time automated hybrid tools.

[0037] In another embodiment, the provided concepts may include production-side tools such as an audio mixer or audio processor adapted for manual operation, a plug-in, or a stand-alone software.

[0038] In an embodiment, the speaker setup may, for example, conform to an international audio multi-channel standard. For example, it may be expected that the speakers are placed on the surface of a sphere, while the listener may, for example, be located at the center of the sphere, and therefore the distance between each speaker and the listener may be equal. In another embodiment, if this is not the case, for example, the levels and propagation times of the speakers may be assumed to be equal in order to achieve the same setup as described above. In other words, even though the physical distance to the listener may be different, such equalization may, for example, compensate for distortion.

[0039] In another embodiment, a speaker setup dedicated, for example, to the reproduction of only a background signal may include only one audio signal located at any point in space.

[0040] Hereinafter, embodiments of the present invention will be described in more detail with reference to the accompanying drawings, in which:

[0041] Fig. 1a shows an apparatus for generating one or more audio output channels from one or more audio signals according to an embodiment.

[0042] FIG. 1 b shows an apparatus for encoding one or more audio signals in a bitstream according to an embodiment.

[0043] FIG. 1 c shows an apparatus for acquiring gain information according to another embodiment.

[0044] Figure 2a shows an arrangement in which a single ducking gain is applied equally to all channels of the background signal.

[0045] FIG. 2b shows an arrangement according to an embodiment, wherein a per-channel scaling value is applied to a single ducking gain.

[0046] Figure 3 A block diagram of an MPEG-H system according to an embodiment is shown, wherein gains according to an embodiment are applied.

[0047] Figure 4 A table showing a number of different source test items comprising a listening test is shown.

[0048] Figure 5 A table is shown illustrating the different weights applied to the channel groups in the first listening test (Listening Test, Part A).

[0049] Figure 6 Shown is a table with multiple speaker configurations employed in listening tests.

[0050] Figure 7 The results of a first listening test are shown.

[0051] Figure 8 A table is shown illustrating the different weights applied to the channel groups in the second listening test (Listening Test, Part B).

[0052] Fig. 9 The results of the second listening test are shown.

[0053] Fig.10 Shown are the loudspeaker placement and gains for a particular calculation example with great circle weighting.

[0054] Fig.11 An egg-shaped weighted loudspeaker layout and gains are shown for a specific calculation example.

[0055] Fig.12 Examples of scaling factors up to 22.2 are shown for several layouts.

[0056] Fig. 1a shows an apparatus for generating one or more audio output channels from one or more audio signals according to an embodiment.

[0057] The apparatus comprises a gain determiner 110 configured to generate or receive gain information, wherein the gain information indicates, for each of at least one audio signal among one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position.

[0058] Furthermore, the apparatus comprises a signal generator 120 configured to generate one or more audio output channels from the one or more audio signals by attenuating or amplifying the at least one audio signal according to the gain information.

[0059] In an embodiment, the gain determiner 110 may, for example, be configured to generate the gain information according to a position associated with each of the at least one audio signal and a reference position.

[0060] According to an embodiment, the gain determiner 110 may, for example, be configured to receive gain information from other devices.

[0061] In an embodiment, the gain determiner 110 may, for example, be configured to receive gain information in a bitstream.

[0062] According to an embodiment, the signal generator 120 may, for example, be configured to generate one or more audio output channels by mixing (e.g., by summing, or e.g., by weighted summing) one or more audio signals after at least one of the one or more audio signals is attenuated or amplified.

[0063] In an embodiment, the gain information may be, for example, gain correction information. The gain determiner 110 may, for example, be configured to receive initial gain information and to generate corrected gain information from the initial gain information using the gain correction information. The signal generator 120 may, for example, be configured to generate one or more audio output channels from one or more audio signals by applying the corrected gain information to at least one audio signal.

[0064] According to an embodiment, the apparatus of Fig. 1a may for example comprise a signal decoder, for example an MPEG decoder, for example an MPEG-D decoder, or for example an MPEG-H decoder, or for example an MPEG-I decoder.

[0065] Fig. 1b shows an apparatus for encoding one or more audio signals in a bitstream according to an embodiment.

[0066] The apparatus comprises a gain determiner 160 configured to generate gain information, wherein the gain information indicates, for each of at least one audio signal of the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position.

[0067] Furthermore, the apparatus comprises a bitstream generator 170 configured to encode one or more audio signals in a bitstream, wherein the bitstream generator 170 is configured to generate the bitstream such that the bitstream comprises the gain information.

[0068] FIG. 1 c shows an apparatus for acquiring gain information according to another embodiment.

[0069] The device comprises a user interface 165, the user interface comprising an input device enabling a user to input input information, wherein the input information comprises gain information, or the device is configured to derive the gain information from the input information, wherein the gain information indicates, for each audio signal of at least one audio signal among the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position.

[0070] According to an embodiment, the apparatus may, for example, further comprise a bitstream generator configured to encode the one or more audio signals in a bitstream, wherein the bitstream generator is configured to generate the bitstream such that the bitstream comprises the gain information.

[0071] In another embodiment, the apparatus may, for example, further include a signal generator configured to generate one or more audio output channels from the one or more audio signals by attenuating or amplifying at least one audio signal according to the gain information.

[0072] In an embodiment, the reference position may be a physical or virtual position of a speaker in an environment.

[0073] According to an embodiment, the reference position may, for example, be associated with an audio signal of the one or more audio signals.

[0074] In embodiments, the apparatus of FIG. 1 b may, for example, comprise a signal encoder, such as an MPEG encoder, such as an MPEG-D encoder, or such as an MPEG-H encoder, or such as an MPEG-I encoder.

[0075] According to an embodiment, the reference position may be, for example, an actual playback position or an assumed playback position of a loudspeaker for playing back an audio signal of one or more audio signals or an audio signal derived from the audio signal. For each of the at least one audio signal, the position associated with the audio signal may be, for example, an actual playback position or an assumed playback position of a loudspeaker for playing back the audio signal or an audio signal derived from the audio signal. Or, for example, the position and / or the reference position associated with the audio signal of the at least one audio signal may be, for example, any physical or virtual point in the 3D space around the listener.

[0076] In an embodiment, the gain information may indicate, for example, for each of at least one audio signal among the one or more audio signals, how to attenuate or amplify the audio signal according to a distance between a position associated with the audio signal and a reference position.

[0077] According to an embodiment, the gain information may, for example, indicate for at least one of the at least one audio signal that the one of the audio signals should be attenuated less or should be amplified more if, compared to another audio signal of the at least one audio signal, a first distance between a position associated with the one of the at least one audio signal and the reference position is greater than a second distance between a position associated with the other one of the at least one audio signal and the reference position.

[0078] In an embodiment, the one or more audio signals may be, for example, two or more audio signals, and the at least one audio signal may be, for example, at least two audio signals.

[0079] According to an embodiment, the gain information may indicate, for example, how to attenuate or how to amplify the audio signal, respectively, for each of the at least two audio signals.

[0080] In an embodiment, the gain information may indicate, for example, for each of the at least two audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position, respectively.

[0081] According to an embodiment, the gain information may indicate, for example, for each of the at least two audio signals, how to attenuate or how to amplify the audio signal according to a distance between a position associated with the audio signal and a reference position.

[0082] According to an embodiment, the gain determiner 110, 160 can, for example, be configured to determine the gain information such that the gain information, for example, indicates the gain information for each of the at least one audio signal, and such that, for example, the sum of the energies of each of the at least one audio signal after attenuation or amplification corresponds to or is estimated to correspond to a reference energy, or is proportional to the reference energy or is estimated to be proportional to the reference energy.

[0083] In an embodiment, the gain determiner 110, 160 may, for example, be configured to calculate or estimate the reference energy as the energy of the at least one audio signal after attenuation or amplification, wherein each of the at least one audio signal is attenuated or amplified with a global gain or a scaled global gain, the scaled global gain being scaled using a scalar value. For example, the gain determiner 110, 160 may be configured to calculate or estimate the total energy of the at least one audio signal as the energy of the at least one audio signal after attenuation or amplification, the attenuation or amplification using a per-channel gain, i.e., a single gain for each of the at least one audio signal.

[0084] According to an embodiment, the gain determiner 110, 160 can, for example, be configured to determine the gain so that the total energy of the at least one audio signal after attenuation or amplification corresponds or is estimated to correspond to a reference energy, or is proportional to the reference energy or is estimated to be proportional to the reference energy by adopting a loss function.

[0085] In an embodiment, the loss function may be defined as follows:

[0086]

[0087] Where l(s) is the loss function, where p represents a real number not equal to 0, where t represents a time index, that is, time index T1 or time index T2 or a time index between T1 and T2, where e ref (t) represents the total energy, where e ref (t) represents the reference energy, where a is a real number. s can be, for example, a real number. s can be, for example, a number described in equation (9) and / or equation (11).

[0088] According to an embodiment, the gain information indicates, for each of the at least one audio signal, how to attenuate or how to amplify the audio signal according to an angle between a position associated with the audio signal and a reference position.

[0089] In an embodiment, the gain information comprises a per-channel gain (e.g., a decibel gain) of each of the at least one audio signal. To generate one or more audio output channels, the signal generator 120 may, for example, be configured to apply, for each of the at least one audio signal, a per-channel gain of the at least one audio signal on the at least one audio signal.

[0090] According to an embodiment, the position associated with each of the at least one audio signal and the reference position may be located, for example, on a circle or an ellipse or on a three-dimensional area around the listener.

[0091] In an embodiment, the gain information indicates, for each of the at least one audio signal, how to attenuate or how to amplify the audio signal according to a distance, such as a great circle distance or other kind of distance, between a position associated with the audio signal and a reference position.

[0092] According to an embodiment, the gain determiner 110, 160 may, for example, be configured to generate and quantize the gain information. Alternatively, the gain determiner 110, 160 may, for example, be configured to receive the gain information as quantized information.

[0093] Regarding some terms, an audio signal (or simply signal) may be a representation of a sound wave. For example, an audio channel (or simply channel) may be an audio signal reproduced by a physical speaker or a virtual speaker located at a specific virtual or physical point in space. For example, channel-based audio may be a format, for example, which may only carry information about the nature of the audio signal and may be reproduced without rendering by a corresponding one or more speakers. For example, object-based audio may be a format, for example, carrying both the nature of the audio signal and metadata required for rendering the audio signal, and reproduced by one or more speakers. Spatial audio may be, for example, a sound signal that reaches a listener from any position in a three-dimensional space. For example, multi-channel audio (or multi-channel audio) may be an audio content comprising two or more separate audio signals. For example, immersive audio (or immersive sound) may be an audio content that, when properly reproduced, may immerse a listener in a three-dimensional sound field. Channel-by-channel processing may, for example, be an operation applied separately to each audio channel of a multi-channel or immersive content. A foreground signal (or simply foreground) may be an audio content or a sound source, for example, an audio content or a sound source that is important for the delivery of the main information or the narration of the story. For example, a background signal (or simply background) may be an audio content or sound source that is less important than a foreground signal. If the level of the background signal may reduce the intelligibility of the foreground signal, the level of the background signal should be reduced.

[0094] Some background considerations are presented below.

[0095] g(t) in equation (1) represents a single global time-varying ducking gain, which can be, for example, a value in the decibel domain, which means that any attenuation factor can be expressed, for example, as a negative value, and the linear gain g lin The relationship between (t) and the decibel threshold g(t) is:

[0096] g(t)=20log 10 g lin (t). (1)

[0097] The unit of g(t) is dB (g(t) is in the decibel domain).

[0098] g lin (t) = 10 g(t) / 20 . (2)

[0099] g lin (t) represents g(t) in the linear domain.

[0100] The output after mixing with ducking gain is:

[0101] y i (t) = f i (t)+b i (t)glin (t). (3)

[0102] Equation (3) describes the scale-free ducking, where f is the foreground signal, b is the background signal; and i is the channel index.

[0103] While a single ducking gain is economical from a computational and transmission perspective, it is not optimal from the aesthetic perspective of the resulting audio scene. To illustrate this point, consider an immersive rain scene with dialogue at or near front center. Attenuating all of the background channels with the same ducking gain will result in a balanced dialogue-to-background ratio, with the overall mix sounding quite good, but that won’t be until the listener moves their head, even slightly. In fact, head movement allows the hearing system to resolve the front / rear distribution of sound sources. As long as the background is attenuated during the active dialogue at front center, the hearing system will notice the attenuation of background sounds coming from the back. While this attenuation may be necessary for the overall dialogue-to-background ratio, the attenuation of the rear channel signals will result in a “holey” experience, as the dialogue signal cannot make up for the energy lost by ducking. The denser the rear background audio, the more prominent the hole.

[0104] Specific embodiments of the present invention are described in detail below.

[0105] According to some embodiments, the problem of holes appearing in unshielded areas of the audio scene (and more generally in any channel other than front center) is addressed.

[0106] In some embodiments, the amount of ducking that reproduces signals in channels that are farther from the foreground position can be reduced, for example. In extreme cases, ducking can be completely disabled for the farthest channels.

[0107] According to some embodiments, this can be achieved, for example, by scaling the channel by a factor c i To achieve this, for example, it can be applied to linear ducking gain to obtain linear gain per channel, such as:

[0108] g lin,i (t) = c i g lin (t) (4)

[0109] Alternatively, in a preferred embodiment, the per-channel scaling factor c i Can for example be applied to decibel scaled gain like:

[0110] g i (t) = c i g(t). (5)

[0111] c i is the scaling factor applied to g per channel.

[0112] In certain embodiments, the transmission standard of MPEG-DDRC may be used, which standard, for example, supports defining scaling values ​​for each channel, which may be applied to decibel scaled ducking gains according to embodiments.

[0113] In the embodiment, c i The value of can be designed manually, for example, to achieve the highest overall aesthetic quality. However, in other embodiments, c i The value of can, for example, be determined automatically. The large number of possible loudspeaker layouts and different personal tastes are reasons for automated design.

[0114] In an embodiment, the distance between the foreground speaker positions and the background speaker positions may, for example, serve as a basis for the scaling.

[0115] Some embodiments are based on the concept that the larger the distance, the more each channel scales c i Should be smaller to reduce ducking depth for these channels.

[0116] In certain embodiments, the distance between the foreground speaker positions and the background speaker positions, such as the great circle distance or other types of distances, can be used as the basis for scaling. The great circle distance is the shortest distance between two points on the surface of a sphere. It should be noted that the use of the great circle distance assumes that the sensitivity of the auditory system is the same in all three axes, which is not the case in reality. However, the main benefit of this measurement is that it is easy to calculate in closed form. If an ellipsoid is used instead of a sphere, there is no longer a closed form solution, and a more computationally expensive numerical approximation is required. In other words, using the great circle distance can provide computationally efficient and suitable quality results.

[0117] For example, some embodiments may use the distance between two sound sources (foreground and background), with the use of the great circle distance being just one of many different embodiments that may represent the calculation.

[0118] Two points p 1 =[x 1 ,y 1 , z 1 ] T and point p 2 =[x 2 ,y 2 , z 2 ] T (Cartesian coordinates) (both are in [0, 0, 0] T The distance between the great circles on the unit sphere with the center being ( T represents a transpose, such as a matrix transpose or a vector transpose: used to indicate that three scalars in an equation should be represented vertically), and for example the listener can be assumed to be at the center of a sphere, such as ||pi ||=1, (the size and length of vector p should be equal to 1. Because it can be assumed that all points are located on the unit sphere). For example, the great circle distance can be defined according to the angle between the two vectors.

[0119]

[0120] Among them (p i , p j ) represents the inner product between two vectors (matrix calculation / vector calculation). When the two points coincide, the distance value obtained is the minimum value; when the two points are on opposite sides of the sphere, the distance value obtained is the maximum value.

[0121] Now suppose the main prospect is located at p 0 , the background channel i is located at p i In an embodiment, the per-channel scaling may be obtained, for example, by

[0122] c i =1-d(p 0 , p i ). (7)

[0123] (p 0 and p i The larger the distance between them, the i The smaller. )

[0124] In an embodiment, the scaling value may be further adjusted if necessary, for example to limit the dynamic range of the position-based scaling. For example, without further scaling and applying the decibel domain value, there will not be any ducking at the speakers behind the listener (assuming front center foreground). This may also be aesthetically undesirable. The two values ​​r may be lo and r hi The distance between them can be limited, for example by:

[0125] c i,lim (p 1 , p 2 )=r lo +c i (r hi -r lo ). (8)

[0126] For example, rl o = 0.3 and rhi = 1. limm stands for limit or scale, meaning "further scaled c i ”.

[0127] The range defines how much scaling to apply to the dodge. When the range is 0-1, full scaling is applied. When the range is 0.5-1, half scaling is applied. lo and rhi The definition of the difference between , in the range 0-1, is not very relevant, since it is the multiplicative coefficient applied to the scaling of individual i. The second stage of the process is more relevant.

[0128] In practical applications, it can be assumed, for example, that the main foreground sound is a point source located at a fixed position, for example, matching the front-center-center channel (azimuth=0, elevation=0). i and limit the zoom value c i，lim Can be used interchangeably. i All embodiments described, for example, can equally use c i,lim Instead of c i to achieve, and vice versa.

[0129] As an alternative to the above embodiment, in other embodiments, for example, each channel scaling c can be manually defined. i .

[0130] In this way, the problem of holes in the acoustic scene that are far away from the foreground sound location can be further resolved.

[0131] Figure 2a shows an arrangement that applies a single ducking gain equally to all channels of the background signal to obtain a ducked background. This results in a perceived hole in the audio scene, with the hole becoming larger the further away from the foreground signal location.

[0132] Fig. 2b shows an arrangement according to an embodiment, wherein a per-channel scaling value is applied to a single ducking gain.

[0133] In an embodiment, these scaling values ​​may be transmitted, for example, in a bitstream.

[0134] In another embodiment, these scaling values ​​may be provided externally, for example.

[0135] Scaling is applied to each channel to obtain a per-channel ducking gain, which is then applied to the background signal to obtain a spatially scaled ducking of the background signal, solving the perceptual hole problem.

[0136] The reference to MPEG-H demultiplexer (demultiplexer) in FIGS. 2a and 2b should be understood as being only a non-limiting example. Other embodiments involve other demultiplexers not related to MPEG-H. Further embodiments do not employ a demultiplexer at all.

[0137] The scaling of the ducking gain according to the spatial position of the reproduced channel position solves the problem of sound scene holes. However, it also introduces new problems. The overall ducking amount in the background signal energy has changed from the original version (all channels are ducked equally). For example, setting r lo <1, r hi= 1, the result is usually less ducking than the original case. Even though spatial separation between background and foreground may alleviate the problem to some extent, the amount of ducking in the overall mix may still be too small, for example, when the foreground is active, the background signal may be too large.

[0138] In some embodiments, a signal and / or layout dependent additional global scaling s may be applied, for example, to the per-channel ducking gain scaling values:

[0139]

[0140] In an embodiment, ("c capped") denotes a modified value of c, the value s may, for example, be independent of the channel. s may, for example, be a global value, applied to every per-channel value of every i.

[0141] To determine the value s, assume that the background signal level per channel is e i (t). In particular, e i (t) can be assumed to represent the energy value of channel i at time t, for example.

[0142] In the embodiment, e i (t) may be, for example, the actual signal energy level. i (t) may be, for example, the measured signal energy level of channel f. Or, for example, in another embodiment, e i (t) may be, for example, the estimated signal energy level of channel i.

[0143] In a further embodiment, e i (t) may be a symbol weighting value. The symbol weighting value may be used, for example, by not measuring the on-time energy of channel i, but by assigning a constant weighting coefficient to channel i.

[0144] The total ducking of the background signal energy at time t is (using the reference ducking value equally in all channels)

[0145] e ref (t)=∑ i e i (t)10 g(t) / 10 . (10)

[0146] The total background energy after ducking using the globally scaled per-channel values ​​is

[0147]

[0148] The goal now is to make the two energies in equations (10) and (11) equal, since the overall ducking for a multi-channel layout using scaling is intended to be equal to the ducking without scaling.

[0149] In order to make the total energy of the two methods of dodging background signals equal, it can be expressed as:

[0150] e ref (t)==e scaled (t). (12)

[0151] In other embodiments, for example, it may not be necessary to satisfy e ref (t)==e scaled (t).

[0152] For example, a factor can be applied to the reference energy e ref (t), for example:

[0153] p·e ref (t)==e scaled (t). (12a)

[0154] For example, p is a real number not equal to 0.

[0155] For e defined in equation (12) ref (t)==e scaled (t), unfortunately, this equality requirement cannot be satisfied (in other words: solved) for non-trivial energies and a wider range of dodge gain values. However, the equality requirement can be reformulated in the form of a loss function, such as:

[0156]

[0157] In the embodiment where a=2, the result of equation (13) is equation (14)

[0158]

[0159] The loss function of (13) is expressed as e ref and e scaled The deviation between .

[0160] Or, if the reference energy e is ref (t) is applied as a factor, equation (13) becomes:

[0161]

[0162] In general, if p=1, equation (12a) is equal to equation (12), and equation (13a) is equal to equation (13).

[0163] In particular, the loss function minimizes the variance of the total energy obtained at all times between the reference background level and the modified background level. In other embodiments, other forms of loss functions may be used, for example. Further embodiments do not use a loss function, but instead solve the equality requirement (12) or requirement (12a) in a different way.

[0164] In an embodiment, for a given layout, energy, initial per-channel scaling value, and ducking gain, finding the scaling s that minimizes l(s) can be achieved by analytically solving the equation.

[0165] However, in another preferred embodiment, finding the scale s can be done, for example, by using s init = 1 to initialize the global scaling and then perform a numerical search to achieve it, e.g., using gradient descent to minimize the defined loss function.

[0166] In further embodiments, a grid search may be performed, for example, over an expected range of scaling values. For example, when the initial scaling only reduces the ducking depth, a reasonable search range is 1...2, and the selected solution is the parameter value that provides the minimum loss.

[0167] If the final reproduction layout is different from the transmission layout, the format conversion weight m j，i can be included, for example, to define a (passive) mixing weight from the transmission layout channel i to the reproduction layout channel j. Such additional weights can, for example, be included in the estimation of the resulting ducking energy, for example:

[0168]

[0169] In the above-described embodiment, for example, the scaling can be calculated for each content, since it depends on the integrated energy of the signal. In the embodiments described below, the method can be simplified, for example, by introducing a constant energy.

[0170] The embodiment depends on the energy per channel e i (t) and each channel ducking gain g i (t). Both are signal dependent and time varying. In practice this means that a global scaling factor will be calculated for each content item. Thus, it is guaranteed that the scaling factor is optimal for the specific item and the ducking gain that has been calculated. The potential disadvantage is that this optimization needs to be done for each item, which introduces computational complexity.

[0171] Hereinafter, embodiments in which efficiency is improved through simplification are described.

[0172] According to the embodiment, as a first simplifying assumption, for example, it can be assumed that the energy per channel does not vary with time, but can be assumed to be a constant related to the channel, that is, e i(t) = e i For example, the value could be the average signal energy of the item, or a weighting factor that depends on the spatial position of the channel, or something similar.

[0173] In an embodiment, for example, it can even be assumed that the energy of each channel is equal, that is, e i (t)=w, where w is a constant, for example, w=1.

[0174] According to an embodiment, instead of using the real time-varying ducking gain g(t) in the calculation, for example, a range of "interesting" ducking gain values ​​can be defined, for example, to which values ​​should be scaled and only these values ​​can be optimized. For example, each of the per-channel energy values ​​can be multiplied by each possible gain value, and / or all these combinations can be optimized. In a preferred embodiment, for example, such an approach can be adopted, and the ducking gain can be assumed, for example, to be in the range of -20≤g(t)≤0, for example, with 100 linearly spaced values ​​in between.

[0175] In an embodiment, the time-invariant per-channel energy and the manually defined ducking gain range can be combined, for example, to allow the removal of time-dependent variables from the optimization. For example, the optimization can only consider all possible ducking gain values ​​for a constant per-channel energy. Compared to using full degrees of freedom, the number of values ​​that need to be considered in the optimization task is greatly reduced.

[0176] Although these simplifications introduce some degree of numerical error to the overall scaling values, the practical impact is negligible and significant efficiency benefits can be achieved.

[0177] In some embodiments, further efficiency advantages may be achieved, for example, by possible quantization of the transmitted scaling values.

[0178] For example, for applications in MPEG-D and MPEG-H, in some embodiments, the per-channel scaling value For example, it can be quantized, for example for storage or transmission in a bitstream. For example, a quantization step size of 0.125 can actually be used. This can be ignored during the optimization process and only the values ​​produced by the proposed algorithm are quantized. An alternative approach could be, for example, to incorporate quantization knowledge into the optimization task, for example by modifying formula (11) to

[0179]

[0180] Where quant(x) is a function that quantizes a given parameter.

[0181] Quantization can, for example, create regions where small changes in the parameter s do not result in changes in the loss function. This can cause problems for some optimization algorithms, such as those that rely on gradient descent. Other embodiments employ other optimization methods, such as grid search, that are robust to these effects.

[0182] To summarize some of the above embodiments, some embodiments provide a solution for adjusting a single background ducking gain for a single channel of an immersive playback layout, such that the generated ducking avoids perceptual holes of excessive or unnecessary ducking in channels far from foreground signal locations.

[0183] Certain embodiments determine per-channel scaling values ​​based on spatial distance from a hypothesized foreground position and / or apply a global scaling on top of the per-channel scaling values ​​to achieve an overall ducking depth comparable to that achieved if a single global ducking gain were applied.

[0184] The result of the proposed concept can be parameterized, for example, as a list of per-channel gain scaling values ​​that are multiplied by the global ducking gain when per-channel ducking is applied. This allows for efficient transmission of per-channel ducking gain in the form of a single scalar per channel in addition to the global ducking gain, while achieving more pleasing ducking in immersive playback scenarios.

[0185] Further embodiments are described below.

[0186] According to an embodiment, a speaker layout method for optimizing mixing of primary audio signals (eg, dialogue, foreground) and secondary audio signals (eg, background) is provided.

[0187] For example, the term "loudspeaker layout" may be understood as the geometric / spatial position of one or more loudspeakers used to reproduce a signal. If a loudspeaker layout has only one loudspeaker, it will reproduce both foreground and background signals simultaneously, with full ducking, regardless of spatial position.

[0188] Some embodiments are applicable to automatic audio mixing processors, encoders and / or decoders, and / or dynamic audio processors (hardware or software, plug-in or stand-alone), and / or manually operated audio mixing devices (mixing consoles, audio routers).

[0189] According to an embodiment, the concept of a loudspeaker layout that optimizes the mixing of the primary audio signal and the auxiliary audio signal is based on a ducking gain signal that specifies the attenuation (and / or amplification) to be applied on the auxiliary signal.

[0190] In an embodiment, the concept of the speaker layout for the optimized mix may be determined, for example, based on the content, level or other properties of the auxiliary audio signal and / or the primary audio signal.In another embodiment, the concept of the speaker layout for the optimized mix may be artistically designed.

[0191] According to an embodiment, the mixing of the main signal and the auxiliary signal may be optimized, for example, for a specific reproduction loudspeaker layout.

[0192] In an embodiment, the optimization may be performed, for example, based on the spatial physical position or virtual position of one or more speakers in the layout.

[0193] According to an embodiment, the ducking gain applied on the auxiliary signal channel may, for example, depend on a scaling factor.

[0194] In an embodiment, the scaling factor may eg depend on the spatial physical position or the virtual position of the loudspeaker of the auxiliary signal.

[0195] According to an embodiment, the scaling factor may, for example, depend on the distance between the auxiliary signal loudspeaker position and the position of the reference point (eg the main audio signal loudspeaker position).

[0196] In an embodiment, the distance may be measured using, for example, the angle between the speaker and the primary audio source location.

[0197] According to an embodiment, a global scaling term may be employed, for example, for calculating the overall ducking depth.

[0198] In an embodiment, the scaling factor may reduce the magnitude of the correction applied to the auxiliary audio signal channel when the distance between the speaker position of the auxiliary audio signal channel and the position of the reference point (eg, the position of the primary audio source) is greater.

[0199] According to an embodiment, the ducking applied on a specific channel of the auxiliary signal may, for example, depend on a common ducking gain signal and a per-channel scaling value.

[0200] In an embodiment, the per-channel scaling value may be determined based on, for example, a spatial physical position or virtual position of the auxiliary signal speaker, or a distance between a reference point (e.g., a physical position or virtual position of the main audio signal speaker) and a physical position or virtual position of a channel of the auxiliary audio signal speaker.

[0201] According to an embodiment, the global scaling term may, for example, be determined such that after applying the per-channel scaling, the total (instantaneous) energy of the auxiliary audio signal is approximately equal to or proportional to the total (instantaneous) energy of the auxiliary audio signal when only the global mixing gain is applied. For example, this may depend on the level / energy of the auxiliary signal in each channel. For example, the global scaling term may be time-varying and / or may, for example, depend on a ducking gain that varies over time.

[0202] In an embodiment, the calculated per-channel scaling values ​​may be transmitted, for example, as additional side information.

[0203] According to an embodiment, the calculation of the per-channel scaling values ​​may be done at decoder / rendering time, for example when the output speaker layout is known (eg, instead of using the transmit layout).

[0204] In an embodiment, the calculation may, for example, take into account audio channel format conversions that occurred during reproduction when the transmission channel layout differs from the actual reproduction channel layout.

[0205] According to an embodiment, the gain is applied to the MPEG-H system. Figure 3 FIG. 1 shows a block diagram of an MPEG-H 3D audio decoder according to an embodiment. The embodiment is intended to produce an MPEG-H audio scene. Figure 3 The operation of the MPEG-H Audio decoder is described in the block diagram of FIG.

[0206] Ducking can be applied, for example, as "DRC-1". This means that it can be applied to the complete object representation before format conversion. For example, when playing a 22.2 bed and mono dialogue in a 5.1 system, the ducking gain can be applied, for example, on the 22.2 bed signal, downmixed to 5.1 and then mixed with the dialogue object. The net effect of this is that no information is available in the reproduction layout when ducking is applied and the ducking is always the same.

[0207] In the following, the development and testing of specific embodiments are described.

[0208] A series of listening tests were conducted to gather evidence to determine which methods, configurations and settings are particularly suitable for applying scaled gain sequences to multi-channel beds and to develop and demonstrate the effectiveness of the proposed solution.

[0209] The main objectives of these studies are to investigate spatial sound perception with a focus on multichannel background ducking, to define the minimum and optimal channel grouping suitable for scaling ducking, to specify multichannel background ducking parameters to achieve pleasant and balanced spatial scenes, to define scaling weighting values, and to avoid dynamic gain as much as possible to save bitrate / complexity.

[0210] These studies are based on the non-limiting exemplary assumptions that the foreground portion is always mono and reproduced by the center loudspeaker only, and that the channel grouping is always performed along the frontal axis, both in the horizontal and vertical planes, to keep the left / right and top / bottom balance unchanged.

[0211] Preliminary testing has been conducted. The test design described below is based on the results of this preliminary testing. Preliminary testing involved listening to candidate content in various layout formats with the goal of identifying the material that would best support adaptation and gaining an initial understanding of the problems and possible solutions.

[0212] Results from preliminary testing indicated that there was no benefit to using both male and female voices in each item. It was then decided to keep only the male test content because it covers a larger frequency spectrum and has more pauses during the speech recordings (pauses in the speech content help assess ducking quality). Furthermore, results showed that the top-front channels do not seem to require as much ducking due to the significant separation provided by the distance from the center channel reproducing the foreground (the “cocktail party” effect), and that the perception of the isolated channel group correlates very well with the perception when all channels are active simultaneously. Preliminary testing fully confirmed the effectiveness of the approach and was able to drive meaningful tuning.

[0213] Program excerpts representing typical broadcast content were selected as source test items. In addition, a custom-made immersive decorrelated noise signal was used.

[0214] Figure 4 A table including a plurality of different listening test source test items is shown.

[0215] The listening test for defining channel groupings is described in detail below (Listening Test, Part A). The purpose of this listening test is to define the minimum and optimal channel groupings for applying zoom ducking. Source test item ABAMC00 has been used for all available background formats. The multichannel Bed was divided into five groups. Both foreground and background have been normalized to the same program loudness level (-27LUFS on the ITU-R.BS1770 scale). The configuration with the highest grouping resolution was used. The listening test asked the listeners (sound experts only) to evaluate the intelligibility of the foreground (FG) signal and the overall pleasantness of the mix (assessed using the labeled MUSHRA scale) for each condition.

[0216] Figure 5 The table shows the different weightings applied to the channel groups in the listening test. 1 means ducking is applied and 0 means no ducking is applied. The purpose of the listening test was to show the impact of each of the 5 groups (Front, Center, Rear, Top Front, Top Right), to find the configuration that makes more sense and to reduce the number of groups by combinations (e.g. Front + Center or Front Top + Top Right).

[0217] During the listening test, the subjects were presented with seven control screens, each containing nine conditions. Each condition consisted of an automatically generated mixture of foreground sounds (German speech, male) and background sounds (random noise, power slope -12 dB / oct). The foreground sounds were always reproduced by the central speaker, while the background sounds were reproduced by only one or two symmetrical speakers, depending on the control screen displayed. The seven control screens were randomly displayed, including Figure 6 Speaker combinations in the table shown.

[0218] The nine conditions differ in the ducking scaling factor that is applied. The first condition has a factor of 0 (no ducking). The last condition has a factor of 1 (maximum ducking). From the 2nd to the 8th condition, gradually increasing factors are applied (0.125, 0.250, 0.375, 0.500, 0.625, 0.750, 0.875).

[0219] As a guideline for the first listening test (Listening Test, Part A), subjects were asked to rate the quality of the overall mix on a scale from 0 to 100 (poor to excellent) based on whether the balance between foreground and background was correct and appropriate. They were asked not to take into account the time constant used to generate the automatic mix. Subjects were also provided with the following guidelines: "The clearer the speech (no unnatural, annoying noise masking the speech), the higher the score. If the speech is not easily heard due to too much noise, the score should reflect this and tend towards lower scores. However, if the speech sounds clear but the changes in the noise level are too obvious and easily noticeable (because it is not masked by the speech), the score should be lower. The highest score should mark the best balance between foreground and background and indicate that changes in the noise level are not noticeable or annoying."

[0220] Figure 7 Table showing the results of Listening Test Part A.

[0221] A listening test (Listening Test, Part B) is presented below which defines the optimal scaling weights for each channel group of 7.1.4Bed. The purpose of this listening test is to determine the optimal weights for each group based on the results of the Listening Test Part A (7.1.4 only).

[0222] As the applied method, the configuration with the highest score in Listening Test A was selected. Figure 8 As shown in the table, 01 to 08 test items were used with different scaling weighting settings for the selected configuration. The listening test required listeners (sound experts only) to rate the intelligibility of the foreground signal and the overall pleasantness of the mix (assessed using the labeled MUSHRA scale).

[0223] Figure 8A table illustrating the different weightings applied to the channel groups during Part B of the listening test is shown.

[0224] As expected, from the listening test part A, an indication of the optimum scaling weights for each channel group is expected.

[0225] In listening test part B, subjects were presented with twelve control screens showing real program excerpts. Each control screen consisted of eleven conditions. Each condition consisted of an automatically generated mix of a foreground signal (speech), which was always reproduced by the center loudspeaker, and a background signal (7.1.4 immersive bed), which was reproduced by all loudspeakers. The eleven conditions differed in the way the scaling gain was applied. Specifically, in nine of the eleven conditions, the same overall ducking amount was applied, but the ducking distribution in the immersive layout differed (different per-channel gains). The remaining two conditions used extreme settings and were proposed for validation purposes.

[0226] As a guideline for the second listening test (Listening Test, Part B), subjects were asked to rate the quality of the overall mix based on whether the balance between foreground and background components was correct and appropriate, with a score from 0 to 100 (poor to excellent). It was recommended not to consider the time constant used to generate the automatic mix, but to focus only on how the individual channels of the background components were ducked. Subjects were also provided with the following guidelines. "The more balanced, pleasing and smooth the background attenuation result is, especially considering how the individual channels of 7.1.4Bed are ducked, the higher the score. If the background attenuation is uncomfortably obvious, such as some channels being annoyingly ducked more than others, the lower the score. If multiple conditions are to be scored, feel free to give them the same score. Due to the long duration of the entire listening test, it may be necessary to stop the test after the first 6 or 7 control screens, take a break, and continue the evaluation when you feel comfortable, either on the same day or on a new day."

[0227] Fig. 9 The results of the second listening test (Multi-channel Bed Part B) are shown, showing the average of the 10 listeners and the 95% confidence interval. Fig. 9 As shown, the second listening test (Listening Test, Part B) provided two very important results, namely clearly demonstrating that scaling the ducking gain on a per-channel basis is useful and that the differences between different per-channel settings are small.

[0228] The following describes the algorithm for calculating scaled spatial ducking proposed by the tests. After conducting the first, second and third listening tests, we also considered more aspects. The idea is to expand the ducking method to include layouts up to 22.2 and any speaker position.

[0229] Due to the increasing number of possible configurations, it was decided to adopt a more flexible approach to the problem, which would specifically take into account the precise localization of discrete background sound sources per channel.

[0230] Specifically, two different approaches, namely the “great circle weighting method” and the “egg-shaped weighting method”, are considered, as described below.

[0231] First, we introduce “great circle” weighting. Fig.10 The loudspeaker layout and gains for a specific calculation example of great circle weighting are shown. The points on the X and Y axes are Cartesian coordinates on the horizontal plane, where

[0232] X>0->front, X<0->back

[0233] Y>0->left, Y<0->right

[0234] The Z-axis represents the height of the speaker in the vertical plane.

[0235] according to Fig.10 The concept of "great circle" is used to calculate the great circle distance between the speaker and the assumed dialogue position (C000). This distance can be simplified to the three-dimensional spatial angle between the two vectors corresponding to the dialogue position and the speaker position. The distance or angle is then normalized to the specified scaling range, here 0.3 to 1.0. The significant property of great circle weighting is that the farther the speaker is from the dialogue position, the less it is ducked. The main disadvantage of "great circle" weighting is that it does not take into account the reception characteristics of the ears pointing to +-90 degrees, and the three spatial axes have different relevance to perception.

[0236] Now let's introduce "egg" weighting. Fig.11 Loudspeaker layout and gains are shown for a specific calculation example of egg weighting.

[0237] The points on the X and Y axes are Cartesian coordinates of the horizontal plane, where

[0238] X>0->front, X<0->back

[0239] Y>0->left, Y<0->right

[0240] The Z-axis represents the height of the speaker in the vertical plane.

[0241] Subjective measurements of spatial masking along a horizontal circle indicate that sources directly behind the listener are masked almost as strongly as if the source were in the same location as the masker. There are sophisticated data-driven models for this, but a simplified approach is to distort the unit sphere of speakers so that it is stretched by a considerable amount on the left-right axis and a smaller amount on the up-down axis. The purpose of this is to get a very rough measure of the importance of the three spatial axes. The weighting is calculated by taking the Euclidean distance (e.g., straight-line distance) between each speaker and the dialog position and mapping it to a range of 0.3 to 1.0. The resulting scaling will apply almost full ducking depth to the signal from the speaker behind the listener, and minimal ducking depth to the signal from the speaker directly to the left / right of the listener. This weighting simulates spatial masking, especially if the listener does not move their head. Signals from the rear speakers will be heavily ducked due to front / back aliasing. However, if the listener can move their head, the front / back aliasing problem is resolved, and the result may be perceived as over-ducking the rear speakers.

[0242] The following describes initial testing to determine the preferred method.

[0243] For both the large circle and egg configurations, two automatic ducking settings were used.

[0244] Both methods were tested in separate listening sessions. Comparing the large circle and egg configurations, in general, the background sounded louder with the egg method when ducking. This could be a perception issue, or it could be one that can be measured and compensated for. Compensation would make the comparison between the two methods more reliable. When standing still and looking towards C, the two methods sounded surprisingly similar (apart from the difference in overall background level). When the head moves, the large circle method has a more complete sound scene at the rear: no unnecessary ducking. The rear can be ducked less, and the denser the scene at the rear, the stronger the difference and / or preference. To some extent, especially without turning the head, the two configurations are similar. However, there are some relevant differences. In general, the large circle ducking seems to attenuate the front channels more than the egg-shaped ducking, which is unpleasant. The rear and / or immersive attenuation of the egg-shaped method is unnecessary and results in a soundstage that is unpleasantly lacking in energy. With the egg-shaped method, the center of gravity of the sound moves unnecessarily to the front during a ducking event. This does not happen, or at least is less noticeable, with the large circle method. The above effects are more pronounced when the head is rotated 90 degrees. Regardless of the configuration used, the content seems to play a big role in how the ducking is performed. The overall perception is somewhat dependent on the content.

[0245] For the optimization coefficient of great circle scaling, once it was discovered that the great circle approach solved the first problem of computing per-channel scaling, the solution was further improved to achieve distributed ducking independent of background layout and reproduction layout.

[0246] To validate this idea, we designed and performed the following listening test, where downmix stereo ducking was compared with native stereo ducking. The goal was to validate the per-channel coefficients and find the most statistically valid values.

[0247] As a test design, a stereo downmix of the 22.2bed signal (uncorrelated pink noise) was created and the stereo mix program was processed using the auto-ducking tool.

[0248] The resulting mixed 22.2 signal is downmixed as the stereo downmix program. The stereo upmix program is compared to the stereo downmix program. The per-channel settings are modified, assuming equal energy, until the stereo upmix program and the stereo downmix program sound similar (steps of this paragraph).

[0249] Several scaling settings have been tested. Their aim was to produce a downmix ducking mix that would aesthetically sound as close as possible to the native stereo ducking mix. To arrive at the result of the downmix test, a criterion was adopted where the difference was assessed by summing the phase-inverted downmix ducking mix with the native stereo downmix ducking mix. Only the area where ducking was fully activated was considered. The smaller the resulting sum, the closer the two versions are.

[0250] Test results show that downmixing generates additional overall energy for background channels. This becomes very apparent in the non-ducked sections, where the downmixed signal is often much louder than the equivalent native stereo signal. When ducking is not activated and levels are changed, the problem is outside of the ducking scaling.

[0251] Further considerations for the test are discussed below. It is worth noting that the ducking calculated by the automatic ducking tool defines how much energy we need to suppress from the background signal. From an energy point of view, it is acceptable if all channels are ducked equally, i.e. all channels comprise equal energy. Since ducking all channels equally sounds annoying, it is desirable to redistribute the ducking. For example, gain scaling can be used, e.g. to reduce ducking in the rear. By scaling, some overall ducking may e.g. be lost, while the remaining energy may e.g. still be too high.

[0252] Numerical testing via a simple equation in Python shows that it is impossible to find a global scaling that is applied to the corrected ducking gains so that the total amount of ducking is again the same as before the per-channel scaling. The main problem in the numerical example is that the accuracy of the compensation depends on the value of the ducking gains. The deeper the ducking, the larger the overall scaling factor, which has the same effect on the overall energy of the per-channel scaling as in the case of equal ducking per channel. We note that there is a conceptual problem in applying a global scaling to all ducking gain values: the scaling operates as a multiplier and the output depends on the original ducking gains. Smaller gains are corrected less (negatively) than larger (negative) ducking gains. This means that scaling of the ducking gains cannot replace proper gap settings.

[0253] To address this problem, an approximation method is proposed based on the observation that a single global compensation scaling cannot provide accurate energy level compensation for different ducking gain values. The core idea is that after applying the global compensation scaling, the error should be uniformly distributed over the range of possible ducking gains in some way. The definition of "uniform" is a design parameter, but a reasonable alternative is to expect that after applying the global compensation, the total energy of the loudspeaker should be the same as if no per-channel scaling was applied. This can be used as the loss function for a model that has only one parameter, the global compensation scaling value.

[0254] This optimization task can be solved analytically, for example, but a simpler approach is to use numerical optimization. Since the value is always greater than 1 and is one-dimensional, it may be sufficient to use a straight line search, for example. The first implementation now uses gradient descent.

[0255] Examples of scaling factors are provided below. For several layouts up to 22.2, large circle scaling, and an initial scaling range of 0.5…1, the energy conservation algorithm provides the following Fig.12 Per-channel ducking gain scaling values ​​shown (for a ducking gain range of -20..0 dB).

[0256] Final tests according to a specific embodiment are now described.In order to evaluate the subjective performance of the provided algorithm and the downmix effect occurring in the end-user device, the following listening test was designed and performed.

[0257] 1. Generate a downmix version of the selected layout (mono, stereo, 5.1, 7.1.4) for each 22.2 native content.

[0258] 2. Normalize them to the same loudness level - 24LUFS.

[0259] 3. For each native content, select a background layout from the Selected Background Layout list (Mono, Stereo, 5.1, 7.1.4, 22.2).

[0260] 4. Select a range (0.3…1, 0.5…1) from the list for the selected background layout.

[0261] 5. Apply the corresponding optimized gain coefficients to generate the ducking mix test project.

[0262] 6. Repeat steps 3, 4, and 5 until all combinations are generated.

[0263] 7. For each native content, listen to the ducked mix for each layout in the range of 0.3…1.

[0264] 8. Evaluate the quality of the dodge in terms of dialogue comprehensibility, enjoyability, and consistency of layout.

[0265] 9. Repeat steps 7 and 8 with a range of 0.5…1.

[0266] 10. Add notes to the conclusion.

[0267] 11. For each of the 22.2 “mix ducked” contents, generate 7.1.4, 5.1, stereo and mono downmix versions (called post-ducking downmix).

[0268] 12. For each layout, each downmixed program is compared to the "mixed ducked" content produced with the downmix background (called the pre-ducking downmix).

[0269] 13. Comment and annotate the consistency of the two versions (11. Downmix after ducking 12. Downmix before ducking)

[0270] Final listening test results are as follows: Overall quality is good. The algorithm performs as expected and allows for layout-independent, unassisted automixing. All versions sound easy to understand. While the 0.3-1 range seems to produce a more pleasing distribution of fades in an immersive soundstage, it is difficult to find a more suitable range setting. All layouts sound roughly the same. The pre-ducking and post-ducking downmix versions sound the same in the parts where ducking is active (foreground signal is present). However, the pre-ducking downmix has a better overall experience due to the loudness adjustment of the already downmixed background parts. However, the post-ducking downmix is ​​completely understandable.

[0271] Although certain aspects are described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, where a module or apparatus corresponds to a method step or a feature of a method step. Similarly, each aspect described in a method step also represents a description of a corresponding module or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0272] According to some implementation requirements, embodiments of the present invention may be implemented in hardware or software, or at least partially in hardware or at least partially in software. Implementation may use digital storage media, such as floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs, EPROMs, EEPROMs or flash memory, which store electronically readable control signals that can (or can) cooperate with a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0273] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0274] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, which is operable to perform one of the above methods when the computer program product runs on a computer.For example, the program code can be stored on a machine-readable carrier.

[0275] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0276] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0277] A further embodiment of the inventive method is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) on which is recorded the computer program for performing one of the methods described herein. The data carrier, the digital storage medium, or the recorded medium are typically tangible and / or non-transitory.

[0278] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.For example, the data stream or the sequence of signals may be configured to be transmitted via a data communication connection, for example via the Internet.

[0279] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0280] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0281] A further embodiment according to the invention comprises an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, a mobile device, a storage device, etc. For example, the apparatus or system may comprise a file server for transmitting the computer program to the receiver.

[0282] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the method is preferably performed by any hardware device.

[0283] The devices described herein may be implemented using hardware devices, or using computers, or using a combination of hardware devices and computers.

[0284] The methods described herein may be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0285] The above embodiments are merely illustrative of the principles of the present invention. It goes without saying that modifications and variations of the arrangements and details described herein will be apparent to other persons skilled in the art. Therefore, the present invention is intended to be limited only by the scope of the claims to be filed, and not by the specific details presented herein through the description and explanation of the embodiments.

Claims

1. An apparatus for generating one or more audio output channels from one or more audio signals, wherein the apparatus comprises: A gain determiner (110) is configured to generate or receive gain information, wherein the gain information indicates, for each of at least one audio signal among the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position, and The signal generator (120) is configured to generate one or more audio output channels from one or more audio signals by attenuating or amplifying at least one audio signal according to the gain information.

2. The device according to claim 1, in, The gain determiner (110) is configured to generate gain information based on a position associated with each of the at least one audio signal and a reference position.

3. The device according to claim 1, in, The gain determiner (110) is configured to receive gain information from other devices.

4. The device according to claim 1 or 3, in, The gain determiner (110) is configured to receive gain information in a bitstream.

5. The device according to any one of the preceding claims, in, The signal generator (120) is configured to generate one or more audio output channels by mixing the one or more audio signals after at least one of the one or more audio signals is attenuated or amplified.

6. The device according to any one of the preceding claims, in, The gain information is the gain correction information. wherein the gain determiner (110) is configured to receive initial gain information, and is configured to generate modified gain information from the initial gain information using the gain modification information, and The signal generator (120) is configured to generate one or more audio output channels from one or more audio signals by applying the modified gain information to at least one audio signal.

7. An apparatus for encoding one or more audio signals in a bitstream, wherein the apparatus comprises: a gain determiner (160) configured to generate gain information, wherein the gain information indicates, for each of at least one audio signal among the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position, and A bitstream generator (170) is configured to encode one or more audio signals in a bitstream, wherein the bitstream generator (170) is configured to generate the bitstream such that the bitstream includes gain information.

8. A device for obtaining gain information, comprising: A user interface (165) comprising an input device enabling a user to input input information, wherein the input information comprises gain information, or wherein the device is configured to derive the gain information from the input information, wherein the gain information indicates, for each of at least one of the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position.

9. The device according to claim 8, in, The apparatus comprises a bitstream generator configured to encode one or more audio signals in a bitstream, wherein the bitstream generator is configured to generate the bitstream such that the bitstream comprises gain information.

10. The device according to claim 8, in, The apparatus further comprises a signal generator configured to generate one or more audio output channels from the one or more audio signals by attenuating or amplifying at least one audio signal according to the gain information.

11. The device according to any one of the preceding claims, in, The reference position is the physical or virtual location of the loudspeaker in the environment.

12. The device according to any one of the preceding claims, in, The reference position is associated with an audio signal of the one or more audio signals.

13. The device according to any one of the preceding claims, in, the reference position is an actual reproduction position or an assumed reproduction position of a loudspeaker for reproducing an audio signal of the one or more audio signals or an audio signal derived therefrom, and For each of the at least one audio signal, the position associated with the audio signal is an actual playback position or an assumed playback position of a loudspeaker used to playback the audio signal or an audio signal derived from the audio signal.

14. The device according to any one of the preceding claims, in, The gain information indicates, for each of at least one audio signal among the one or more audio signals, how to attenuate or how to amplify the audio signal according to a distance between a position associated with the audio signal and a reference position.

15. The device according to any one of the preceding claims, in, The gain information indicates, for one of the at least one audio signals, that the one of the audio signals should be attenuated less or should be amplified more than the other one of the at least one audio signal if a first distance between a position associated with the one of the at least one audio signals and a reference position is greater than a second distance between a position associated with the other one of the at least one audio signal and the reference position.

16. The device according to any one of the preceding claims, in, the one or more audio signals are two or more audio signals, and The at least one audio signal is at least two audio signals.

17. The device according to claim 16, in, The gain information indicates, for each of the at least two audio signals, how to attenuate or how to amplify the audio signal.

18. The device according to claim 17, in, The gain information indicates, for each of the at least two audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position, respectively.

19. The device according to claim 18, further according to claim 14, in, The gain information indicates, for each of the at least two audio signals, how to attenuate or how to amplify the audio signal according to a distance between a position associated with the audio signal and a reference position.

20. The device according to any one of the preceding claims, in, The gain determiner (110; 160) is configured to determine gain information such that the gain information indicates gain information for each of the at least one audio signal and such that a sum of energies of each of the at least one audio signal after attenuation or amplification corresponds to or is estimated to correspond to a reference energy or is proportional to the reference energy or is estimated to be proportional to the reference energy.

21. The device according to claim 20, in, The gain determiner (110; 160) is configured to calculate or estimate a reference energy as an energy of at least one audio signal after attenuation or amplification, wherein each of the at least one audio signal is attenuated or amplified by a global gain or a scaled global gain, the scaled global gain being a global gain scaled by a scalar value, and The gain determiner (110; 160) is configured to calculate or estimate a total energy of the at least one audio signal as the energy of the at least one audio signal after attenuation or amplification by a per-channel gain, by a separate gain for each of the at least one audio signal.

22. The device according to claim 20 or 21, in, The gain determiner (110; 160) is configured to determine the gain so that the total energy of the at least one audio signal after attenuation or amplification corresponds or is estimated to correspond to a reference energy, or is proportional to the reference energy or is estimated to be proportional to the reference energy by adopting a loss function.

23. The device according to claim 22, The loss function is defined as follows: Where l(s) is the loss function, where s represents a real number, Where p is a real number not equal to 0. Where t represents the time index, which is the time index T1 or the time index T2 or the time index between T1 and T2. in, e ref (t) represents the total energy, Among them, e ref (t) represents the reference energy, and Where a represents a real number.

24. The device according to any one of the preceding claims, in, The gain information indicates, for each of the at least one audio signal, how to attenuate or how to amplify the audio signal according to an angle between a position associated with the audio signal and a reference position.

25. The device according to any of the preceding claims, further according to any of claims 1 to 6, in, The gain information includes a per-channel gain of each audio signal of the at least one audio signal, The signal generator (120) is configured to apply, for each of the one or more audio signals, a per-channel gain of the one of the at least one audio signals to the one of the at least one audio signals to generate one or more audio output channels.

26. The device according to any one of the preceding claims, in, The position and the reference position associated with each of the at least one audio signal are located on a circle or an ellipse or a three-dimensional area around a listener.

27. The device according to any one of the preceding claims, in, The gain information indicates, for each of the at least one audio signal, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position.

28. The device according to any one of the preceding claims, in, The gain determiner (110; 160) is configured to generate and quantize gain information, or Therein, the gain determiner (110; 160) is configured to receive the gain information as quantized information.

29. The apparatus of any preceding claim, further dependent on any one of claims 1 to 6, wherein the apparatus comprises a signal decoder.

30. Apparatus according to any preceding claim, further dependent on any one of claims 7 to 10, wherein the apparatus comprises a signal encoder.

31. A method for generating one or more audio output channels from one or more audio signals, wherein the method comprises: generating or receiving gain information, wherein the gain information indicates, for each of at least one audio signal among the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position, and One or more audio output channels are generated from the one or more audio signals by attenuating or amplifying at least one audio signal according to the gain information.

32. A method for encoding one or more audio signals in a bitstream, wherein the method comprises: generating gain information, wherein the gain information indicates, for each of at least one audio signal of the one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position, and One or more audio signals are encoded in a bitstream, wherein the bitstream is generated such that the bitstream includes gain information.

33. A method for obtaining gain information, wherein the method comprises: Input information is received via an input device of a user interface, wherein the input information comprises gain information, or wherein the method comprises deriving the gain information from the input information, wherein the gain information indicates, for each of at least one audio signal of one or more audio signals, how to attenuate or how to amplify the audio signal according to a position associated with the audio signal and a reference position.

34. A computer program for implementing the method according to any one of claims 31 to 33 when the computer program is executed on a computer or a signal processor.

Images