Channel Identification of Multi-Channel Audio Signals
The method addresses channel identification issues in surround sound systems by analyzing audio content to detect and correct channel swaps and damage, ensuring accurate spatial audio reproduction.
Patent Information
- Application Number
- JP2022512847
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-10-22
- Filing Date
- 2020-08-27
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-08-27
AI Technical Summary
Current surround sound systems lack functionality for channel identification, leading to swapped or damaged channels that are not detected, resulting in unreliable metadata and compromised spatial auditory impressions.
A method for channel identification in multi-channel audio signals that relies on analyzing the audio content itself to detect empty, LFE, symmetrical, and central channels, using techniques such as energy measurement, spectral distance, and machine learning to restore the correct channel layout.
The method efficiently identifies and corrects channel swaps and damage in real-time, ensuring accurate spatial audio reproduction by relying on audio content analysis rather than metadata, enhancing the reliability and quality of multichannel sound systems.
Smart Images

Figure 0007581333000010 
Figure 0007581333000011 
Figure 0007581333000012
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to PCT Patent Application No. PCT / CN2019 / 103813, filed August 30, 2019, U.S. Provisional Patent Application No. 62 / 912,279, filed October 8, 2019, and European Patent Application No. 19204516.9, filed October 22, 2019, each of which is incorporated by reference in its entirety herein.
[0002] Technical Field The present disclosure relates to the field of channel identification, and in particular to methods, apparatus and software for channel identification for surround sound systems. [Background technology]
[0003] Audio signals are usually transformed several times before reaching a multichannel system. During these transformations, channels can be swapped or damaged. Surround sound processes usually do not include features for channel identification, abnormal channel detection, or channel swap detection, and default layout settings are used. If the channel layout of the input audio data does not match the processing settings, the channels are swapped.
[0004] The current standard is that the swapped channel indexes are stored in the surround sound data as metadata, which is unreliable and harmful for future processes: if the surround sound contains some abnormal channels, the error may go undetected and may lead to further processing.
[0005] Therefore, improvements are needed in this context. Summary of the Invention [Problem to be solved by the invention]
[0006] In view of the above, it is an object of the present disclosure to overcome or mitigate at least some of the problems mentioned above. In particular, it is an object of the present disclosure to provide channel layout identification based on the audio signals of the channels instead of metadata added by the audio codec. This nature makes the identification independent of the encoding format or channel number, making it less susceptible to mismatched metadata. A spatial auditory impression is important for multi-channel surround sound and is typically created by panning sound sources through a mix. The channel identification method described herein extracts spatial information to recover the channel layout. Further and / or alternative objects of the present invention will be apparent to the reader of this disclosure. [Means for solving the problem]
[0007] According to a first aspect of the present invention, there is provided a method for channel identification of a multi-channel audio signal comprising X>1 channels, comprising the steps of: identifying an empty channel, if any, among the X channels, resulting in a subset of Y≦X non-empty channels; determining whether a low frequency effects (LFE) channel is present among the Y channels, and if an LFE channel is present, identifying the identified channel among the Y channels as an LFE channel; splitting the remaining channels among the Y channels that are not identified as LFE channels into any number of channel pairs by matching symmetrical channels; and identifying the remaining unpaired channels among the Y channels that are not identified as LFE channels or split into pairs as center channels.
[0008] By the term "channel identification," in the context of this specification, it should be understood that if the channels of an audio signal are swapped and / or damaged, channel identification may be used to find the correct settings for an audio signal to restore the audio signal to its original intent. The term "channel identification" includes features such as abnormal channel detection and / or channel swap detection.
[0009] By the term "multichannel audio signal" in the context of this specification an audio signal having at least two channels of audio is to be understood. An audio channel is a sequence of audio signals, preferably distinct from at least one other channel of the multichannel audio signal. The audio signal may for example be in the format of an audio file, an audio clip or an audio stream.
[0010] By the term "empty channel" in the context of this document is to be understood a channel of audio having a speech signal content below a certain threshold, which may for example be a total energy content threshold or an average energy content threshold.
[0011] By the term "low frequency effects (LFE) channel" in the context of this specification is to be understood a channel of audio having audio signal content that substantially includes, primarily includes, or only includes energy below a frequency threshold, such as 200 Hz.
[0012] By the term "symmetrical channels" in the context of this specification, channels of audio having sufficiently similar and / or symmetrical audio signal content should be understood. Symmetrical audio signal content may for example include similar background sounds and different foreground sounds, similar base sounds (e.g., low frequencies) and different descant sounds (e.g., high frequencies), respectively, or vice versa. Symmetrical audio content may further include synchronized sounds, such as different parts of a single chord, or sounds that start in one channel and end in another channel.
[0013] By the term "center channel" in the context of this specification should be understood a channel of audio that is substantially independent of other channels, including the most general content of the other audio channels. Although the present disclosure focuses on embodiments having only one center channel, which is the current standard for multi-channel audio signals, as current standards evolve, the method according to the first aspect may be adjusted accordingly.
[0014] The inventors have recognized that identifying the center channel is more difficult than many of the other steps. Thus, computational power can be saved by performing the center channel identification step as the last step in the channel identification method, whereby the computation is reduced to finding the remaining channel after all other channels have been identified, and optionally verifying it as the center channel. Similar efficiencies associated with ordering (i.e., the particular order of steps of the channel identification method described herein) are discussed with respect to specific embodiments, but many of them are generally applicable to most embodiments.
[0015] In addition to saving computational power, the ranking can be used to increase the reliability of the methods by starting with the most reliable methods.
[0016] In a preferred aspect, ranking can be used to both save computational power and increase the reliability of the method.
[0017] According to some embodiments, the method further comprises a step of distinguishing the channels divided into pairs among front pairs, side pairs, rear pairs and / or any other position pairs, the channel pair distinguishing step comprising calculating an inter-pair level difference between every two pairs, the inter-pair level difference being proportional to the decibel difference of the sum of the sub-band sound energies of each pair, and the pair having the relatively higher level being distinguished as the front pair.
[0018] Many multichannel audio signals contain multiple channel pairs, such as 5.1, which includes a front pair and a rear pair. It is therefore beneficial for a method for channel identification to be able to distinguish between position pairs and accurately identify position pairs as such. Inter-pair level difference is an efficient and accurate measure for distinguishing between position pairs.
[0019] According to some embodiments, the channel pair discrimination step further comprises: selecting one or more segments of the signal for each channel of each pair for which the absolute pairwise level difference is above an absolute threshold; and calculating the pairwise level difference of the pair using only those segments, wherein if the relatively highest average pairwise level difference is below the level threshold, the step of calculating the pairwise level difference of the pair is repeated using a higher absolute threshold.
[0020] The level difference between pairs is not always high enough, because differences below, for example, 2 dB may not be useful information. It is therefore beneficial to select segments of the signal whose content may generate larger level differences between pairs. If the selection of a segment does not result in a sufficiently high average inter-pair level difference, a selection with a higher absolute threshold may achieve this.
[0021] Since the absolute pairwise level differences are checked pointwise in these embodiments, the selected segment may contain several isolated frames.
[0022] In other embodiments, absolute values are checked in the segments and the maximum absolute pairwise level difference is compared to an absolute threshold, or the average absolute pairwise level difference is compared to an absolute threshold, whereby the selected segments are quantized by the checked segment length.
[0023] According to some embodiments, if the highest relative average inter-pair level difference is below a level threshold and the absolute threshold is above a maximum threshold, the pair with the highest relative directional coherence is identified as the front pair, where directional coherence is a measure of the similarity of the two channels in the time domain, which is related to the sound image direction, which in turn implies the phase difference between the channels.
[0024] In these embodiments, the selection of segments did not result in a sufficiently high average inter-pair level difference. Therefore, directional consistency is used instead to distinguish the pairs. The pair with the highest directional consistency is distinguished as the forward pair. The signals in the forward pair are usually time-aligned to represent directional sources, so they have high correlation and low delay, and therefore high directional consistency. This means that there are more identical components in the forward pair compared to the rear pair.
[0025] The selection of a segment may fail because the highest average pairwise level difference does not reach a high enough level to exceed the level threshold, and the absolute threshold is so high that the segments above it are not long enough to allow the pairwise level difference to be calculated. If the total length of the selected segments is, for example, shorter than 20% (or any other defined percentage) of the non-silent signal length, or shorter than, for example, 1 minute (or any other defined length), the useful signal may be deemed too short.
[0026] Directional coherence measures the proportion of identical components in a signal by comparing sample values in the time domain at different points. The more similar the signals of the two channels are, the higher the correlation and the smaller the delay. Paired channels usually have correlated signals, and the signals of the forward pair are usually time-aligned to represent directional sources.
[0027] Alternatively, the combined directional consistency with the identified center channel may be used to distinguish the pairs: the pair with the direction closest to the center channel is also closest to the center channel (i.e., the pair is identified as the forward pair).
[0028] According to some embodiments, the empty channel identification step further comprises measuring the sound energy in each channel among the X channels, and if the total sound energy of the channel is below an energy threshold, the channel is identified as empty.
[0029] Sound energy is typically measured using subbands for each channel by summing the amplitude of each frequency in each subband, which provides an efficient way to identify an empty channel even if noise from coding or other factors is present in the empty channel.
[0030] The energy threshold may be, for example, −80 to −60 dB, preferably −70 dB. Instead of or in addition to measuring the total sound energy, the average sound energy in time segments may be measured, where the time segments may be 1 to 10 seconds.
[0031] Empty channels can be the result, for example, of abnormal devices, stereo advertising slots in multichannel TV programs, and multichannel surround sound that has been upmixed from original stereo or mono sound.
[0032] According to some embodiments, an LFE channel is determined to be present among the Y channels if the sum of the subband sound energy in the low frequency region of a channel, any subband below 200 Hz, is significantly greater than the sum of the subband sound energy in all other frequency regions in that channel.
[0033] This is beneficial in that it is less likely to miss the LFE channel. 200Hz is a low frequency cutoff intended to ensure that the LFE channel is not missed while also reducing false positives. Typically the threshold is 120Hz, but since normal channels carry signals over a much wider frequency range it may be preferable to set it to a higher value.
[0034] According to some embodiments, matching symmetric channels in the channel pair splitting step further comprises calculating an inter-channel spectral distance between the channels using the calculated sound energy distribution and variance of each channel, where the inter-channel spectral distance is a normalized pairwise measure of the distance between two matching sound energy sub-bands in each channel, summed over multiple sub-bands; and matching the channels with the shortest distance as a pair.
[0035] The inter-channel spectral distance is a simple and accurate measure of symmetry. Mathematical distance is a measure of similarity that can be weighted in various ways. The distance measures used may be Euclidean distance, Manhattan distance and / or Minkowski distance.
[0036] According to some embodiments, the channel pair splitting stage continues pairing any unpaired channels among the Y channels that are not identified as LFE channels until fewer than two channels remain.
[0037] There may be more than two pairs of channels, such as a forward pair and a backward pair, so if there are more than two channels remaining, there may be more channel pairs between them, and more pairs may be splittable.
[0038] According to some embodiments, the channel pair splitting step further comprises assigning the first received channel of the multi-channel audio signal in each pair as the left channel and the last listed channel in each pair as the right channel. In multi-channel audio signals, it is conventional to list the left channel of each pair before the right channel, so the method is more efficient by assuming that this is always the case.
[0039] According to some embodiments the method further comprises the step of calculating a confidence score for any of the results of the steps of the method, the confidence score being an indication of how reliable the result is, and if the duration of the multi-channel audio signal is below a certain duration threshold the confidence score is multiplied by a weighting factor less than 1, whereby durations below the duration threshold lead to a less reliable result.
[0040] Knowing how reliable the results of each stage of the method are can be useful to diagnose mistakes or measure improvements. If the duration of the multichannel audio signal is too short, the discrimination made is not reliable, since there is too little data available for the calculation. Therefore, weighting factors can be used.
[0041] According to some embodiments, the method further comprises a display step in which the calculated confidence score is displayed on a display, and a warning is displayed if the calculated confidence score is below a confidence threshold and / or if the identified channel layout differs from the user's preferred layout.
[0042] The indication is beneficial in that the user may receive feedback regarding the reliability of the method. This allows the user to make an informed decision as to whether the method's identification is more reliable than the current configuration. The warning is beneficial in that it may alert the user to take action to improve the method, for example by stopping the method, rerunning the method, or by, for example, increasing the bit streaming rate and / or repairing an upstream anomaly. If the identified channel layout differs from the user's configured layout, then the configured and / or identified channel layout may be erroneous, which may require action, for example, by the device or the user.
[0043] According to some embodiments, the method further comprises applying the identified channel layout to the multi-channel audio signal. The applying step may include changing the order of the channels of the multi-channel audio signal; redirecting the channels to the identified playback sources, i.e., so that the left channel is output by the left speaker; or any other physical and / or digital manipulation of the multi-channel audio signal to make it conform to the identified layout that is the result of the present method for channel identification.
[0044] According to some embodiments, the channel layout identified by the method is applied in real-time to the multi-channel audio signal as it is streamed to a speaker system.
[0045] The proposed method is very computationally efficient and can be applied in real-time without significant delay in the playback.
[0046] Initial results may be inaccurate, resulting in a low confidence score, but as more data is collected as the audio signal is played, the score increases.
[0047] According to some embodiments, at least one of the steps of the method uses a machine learning based method, the machine learning based method being a decision tree, Adaboost, GMM, SVM, HMM, DNN, CNN, and / or RNN.
[0048] Machine learning may be used to further improve the efficiency and / or reliability of the method.
[0049] According to a second aspect of the present invention there is provided an apparatus arranged to identify channels of a multi-channel audio signal, the apparatus comprising circuitry arranged to carry out a method according to the first aspect of the present invention.
[0050] According to a third aspect of the present invention there is provided a computer program product comprising a non-transitory computer readable storage medium having instructions adapted to perform a method according to the first aspect of the present invention when executed by a device having processing capability.
[0051] The second and third aspects may generally have the same features and advantages as the first aspect.
[0052] It is further noted that the invention relates to all possible combinations of features unless expressly stated otherwise. [Brief description of the drawings]
[0053] The above and additional objects, features and advantages of the present invention will be better understood through the following illustrative and non-limiting detailed description of preferred embodiments of the invention, with reference to the accompanying drawings, in which like reference numerals are used for similar elements, in which: [Figure 1] 4 illustrates a menu of different surround sound formats according to some embodiments. [Diagram 2]1 illustrates a channel layout for a 5.1 surround sound system, according to some embodiments. [Diagram 3] 1 illustrates a flowchart of an audio broadcast chain according to some embodiments. [Figure 4] 2 illustrates steps in a channel identification method according to some embodiments. [Diagram 5] 2 illustrates steps in a channel identification method according to some embodiments. [Figure 6] 2 illustrates steps in a channel identification method according to some embodiments. [Figure 7A] 4 shows a first half of a flow chart of steps of a channel identification method according to some embodiments. [Figure 7B] 4 illustrates a second half of a flow chart of steps of a channel identification method according to some embodiments. [Figure 8] 1 illustrates a system architecture for a channel sequence detector in accordance with some embodiments. [Figure 9] 2 illustrates steps in a channel identification method according to some embodiments. [Figure 10] 1 shows a flow chart of a channel pair splitting step according to some embodiments. [Figure 11] 1 shows a flow chart of a channel pair position discrimination step according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0054] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the invention are shown. The systems and devices disclosed herein will be described in operation.
[0055] The present disclosure generally relates to the problem of swapped or damaged channels in a multi-channel audio signal. To restore the channels to their intended state, the inventors have found that channel identification can be used. In the following, the multi-channel audio signal is a 5.1 audio signal. However, this is merely an example, and the methods and systems described herein can be used for channel identification of any multi-channel audio signal, such as 7.1.
[0056] Figure 1 shows a schematic diagram of a workstation menu for multi-channel audio processing, with examples of different widely used formats of 5.1 channels.
[0057] The current standard implementation simply involves choosing a default format, and if the channel layout of the input audio data does not match the settings being processed, the channels are swapped. The swapped channel indexes may be stored as metadata in the surround sound data, thereby ensuring correct swapping on an ongoing basis. However, if future systems use different defaults, the metadata will be unreliable and detrimental to future processes.
[0058] If a multi-channel audio signal further contains a damaged channel, current standards do not detect this anomaly and thus the error propagates to future systems.
[0059] Figure 2 shows a typical layout of a 5.1 surround sound system. If the content of any of the speakers in this system is swapped, or if any channel is damaged or left empty, the audio experienced by the listener will differ from the original intent. For example, if the content of the front R and surround R speakers is swapped, the symmetry of the speaker pair is broken, or if the content of the front L speaker is left empty, important parts of the overall sound image may be missing. The sound image of the original surround sound data cannot be reproduced, and the spatial impression becomes disorganized and distracting to the listener.
[0060] An abnormal channel or channels can be detected because their index or entire layout may appear abnormal. Swapped channels, if any, can also be found by comparing the detected channel layout with the channel layout in the user's settings.
[0061] The terms surround pair and rear pair are used interchangeably throughout this disclosure to generalize this disclosure to further possible position pairs, for example in a 7.1 surround sound system the surround pair is replaced by a side pair and a rear pair.
[0062] Figure 3 shows an example of an advanced audio system in a typical broadcast chain. This example shows the flow of surround sound data in a typical broadcast chain, which means that the surround sound is transformed several times during a typical workflow before playback. As mentioned above with respect to Figure 1, metadata errors can propagate through such a workflow. Furthermore, at each process in the workflow, channels can be swapped or damaged.
[0063] The flow begins with a Production, which includes channel-based content, object-based content, and / or scene-based content contributing to advanced sound file formats, which are output by the Production and input to Distribution.
[0064] The distribution includes a distribution adaptation of an advanced sound file format to an advanced sound format, the advanced sound format being output by the distribution and input to the broadcast, and the broadcast including a fork between high bandwidth broadcast and low bandwidth broadcast.
[0065] The low bandwidth broadcast renders the advanced sound format into a legacy stream format that is output by the broadcast and input into the low bandwidth connection / legacy broadcast.
[0066] Low bandwidth connection / legacy broadcast includes direct playback to legacy devices.
[0067] High bandwidth broadcast adapts the advanced sound format into a broadcast stream format that is output by the broadcast and input into the high bandwidth connection / broadcast.
[0068] High bandwidth connection / broadcasting involves any device rendering into a speaker layout or binaural layout for Hi-Fi, TV, phone, tablet, etc.
[0069] Due to the unreliability of metadata, the inventors have found a channel identification method that relies solely on the audio content of the multi-channel audio signal to detect anomalous channels. The detector can detect the layout of channels based on all available data and can further provide an estimated channel index with a confidence score to indicate the reliability. An anomalous channel(s) can be detected because their index or entire layout may appear anomalous. Swapped channels, if any, can also be found by comparing the detected channel layout with the channel layout in the user's settings.
[0070] Typically, the audio data includes a front sound image coming from a center channel and possibly a front channel pair, which maintains directional stability over most of its duration; left and right channels treated as a pair, which convey balanced audio information; and rear channels, which convey information that may enhance the overall sound image. The audio data may further include a separate low frequency channel to plump up the sound image with low frequencies. If multi-channel surround sound accompanies the video or image, the sound image preferably matches the visual image and the designed listening area.
[0071] Basing channel identification on the audio data makes the identification independent of the encoding format or number of channels and insensitive to mismatched metadata. A spatial auditory impression is important for multichannel surround sound and is typically created by panning sound sources through the mix. Channel identification extracts spatial information to recover the channel layout.
[0072] 4 shows a diagram of one embodiment of a channel layout identification method 100. The method 100 includes five steps that are performed in a particular order to minimize required computations.
[0073] The method 100 starts with a multi-channel audio signal containing X>1 unidentified channels. The first step is the empty channel identification step 110 since it is the least computationally intensive step.
[0074] The empty channel identification step 110 involves measuring the sound energy in each channel among the X channels to identify any empty channels, thus resulting in a subset of Y≦X non-empty channels.
[0075] The sound energy in each channel among the X channels may be measured in short-term, medium-term and / or long-term durations and may be measured in the time, spectral, wavelet and / or auditory domains.
[0076] Various terms may be useful depending on the content of the channel.
[0077] The time domain contains information about sound pressure values at different times. The spectral domain contains frequency information in spectral components, arrived at by transforming the channel content. The wavelet domain contains time and frequency information in wavelet multi-resolution decomposition, arrived at by transforming the channel content. The auditory domain is a regular, untransformed domain that contains information about the auditory nerve response evoked by listening to a signal.
[0078] The auditory domain may be used for channel discrimination. For example, a decomposition based on auditory filters, such as Mel / Burk filter banks, may be used in each method step. In such an embodiment, the specific loudness of each critical band is used to replace the subband energies in Eq. 1.
[0079] A wavelet transform can also be applied to the signal decomposition and provide time-frequency features for the following method steps.
[0080] A channel is identified as empty if: the total sound energy of the channel is below an energy threshold, or if the sound energies of the subbands of the channel are below an energy threshold. A subband is a range of energy.
[0081] One definition of subband energy is:
number
[0082] This definition is measured in the short term. For a time block of one or a few frames, E b,c Both the mean and standard variance of (l) are calculated. If both the mean and variance are below certain thresholds for all time blocks, then subband b of channel c is detected as empty.
[0083] Alternatives include band-pass filtered signals and spectrally related measures such as auditory rate maps.
[0084] The identity of the empty channels may be stored using metadata.
[0085] An LFE discrimination step 120 is next and involves determining whether a low frequency effects (LFE) channel is present among the Y channels and, if an LFE channel is present, identifying the discriminated channel among the Y channels as the LFE channel.
[0086] The LFE discrimination step 120 may further include using the sound energy in each of the Y channels measured in the empty channel identification step 110 to determine whether an LFE channel is present, thereby saving computational effort.
[0087] The LFE discrimination step 120 may further include measuring frequency bands in each of the Y channels where sound energy is present above an energy threshold, which does not require the measurement of sound energy in the empty channel identification step 110.
[0088] In each of the Y channels, frequency bands in which sound energy is present above an energy threshold may be measured for short-term, medium-term and / or long-term durations.
[0089] Determining that an LFE channel is present among the Y channels may involve checking whether the sum of the subband sound energy in the low frequency region of the channel is significantly higher than the sum of the subband sound energy in all other frequency regions of that channel, which is beneficial in that the LFE channel is less likely to be missed.
[0090] Instead of summing the subband sound energies, for example, the average and / or maximum values may be used.
[0091] Any such channel may be identified as the LFE channel. The low frequency region may be, for example, 400 Hz, 300 Hz, 200 Hz, 120 Hz, 100 Hz, or any subband below 50 Hz. The low frequency region may be determined based on the content of the audio signal.
[0092] In practice, any frequency between 200 Hz and 2000 Hz can belong to the low frequency region or the high frequency region depending on the embodiment. Thus, the low frequency region can be determined based on the particular embodiment. Alternatively, it may be beneficial to look only at the subbands below 200 Hz and above 2000 Hz.
[0093] The highest frequency of a signal may depend on the sample rate of the signal, so it may be useful to look only at subbands between 2000 Hz and half the sample rate.
[0094] Determining the presence of an LFE channel among the Y channels may include checking whether the channel contains only subband sound energy above an energy threshold in frequency regions below a frequency threshold. This is beneficial in that it is more likely not to detect channels other than the LFE channel, but may not detect the LFE channel if it is, for example, noisy or has a different low frequency region than expected. In some embodiments, only such channels are identified as the LFE channel.
[0095] The frequency threshold may be, for example, 2000 Hz, 1000 Hz, 500 Hz, 400 Hz, 300 Hz, 200 Hz, 120 Hz, 100 Hz, or 50 Hz, or may be determined based on the content of the audio signal.
[0096] If several LFE channels are determined to be present among the Y channels, only one may be identified as the LFE channel according to the hierarchy of the feature or features used to determine whether an LFE channel is present.
[0097] Since most multi-channel audio signals have at most one LFE channel, a hierarchy can be used to determine which of several possible LFE channels is identified as the LFE channel. The hierarchy may, for example, include the maximum difference in subband sound energy between the low frequency region and other frequency regions, or a harder threshold. The identified LFE channel may be stored using metadata.
[0098] A channel pair splitting step 130 follows and involves splitting the remaining channels of the Y channels that are not identified as LFE channels into any number of channel pairs by matching symmetric channels. The channel pair splitting step 130 is discussed further in relation to FIG. 10.
[0099] A center channel identification step 140 is next and involves identifying any remaining unpaired channel of the Y channels that has not been identified as an LFE channel or split into a pair as a center channel.
[0100] The center channel identification step 140 may further include calculating the independence and / or uncorrelation of the remaining unpaired channels of the Y channels that have not been identified as the LFE channel or split into pairs compared to the other channels of the Y channels, and identifying the center channel as the most independent and / or uncorrelated channel.
[0101] This may be calculated, for example, based on measuring the content of different channels in the time, spectral, wavelet and / or auditory domains.
[0102] Of the Y channels, the independence and / or decorrelation of the remaining unpaired channels that are not identified as the LFE channel or split into pairs may only be calculated relative to the channels that are split into pairs, since the center channel is typically the most independent and / or uncorrelated to its paired channels.
[0103] In another embodiment, the center channel identification step 140 is performed after the channel pair discrimination step 150, and the independence and / or decorrelation calculations are only calculated relative to the channels that have been discriminated as forward pairs.
[0104] This is because the central channel is typically the least independent and / or uncorrelated with respect to the forward paired channels, but is still independent and / or uncorrelated. Thus, when independence and / or uncorrelatedness is found, identification of the central channel is very reliable since the chance of false positives is reduced. Comparing the central channel to all pairs is more reliable but more resource intensive.
[0105] Either of these embodiments is beneficial in that they are highly reliable, but may require significant computations, so in one beneficial embodiment, any remaining channels are identified as the center channel without validation.
[0106] If more than one channel remains, all may be identified as the center channel, or an error is assumed and the channel identification method is restarted, either with all steps being redone or only those steps determined to be likely to be in error.
[0107] The repeated steps may be, for example, a free channel identification step 110 and / or an LFE channel discrimination step 120 if an even number of channels remain (as these may provide different parity), or a channel pair division step 130 and / or a channel pair discrimination step 150 if an odd number of channels remain different from one (as these provide the same parity).
[0108] The repeated steps may additionally or alternatively be related to confidence scores of the steps, as will be further described in relation to FIG.
[0109] The identity of the center channel may be stored with the metadata.
[0110] Figure 5 illustrates steps of a channel identification method. This embodiment further includes a display step 160 and an application step 170, which are further described in relation to Figures 8-9, respectively. The sequence shown in Figure 5 is the preferred order because of the efficiency achieved by reusing previous results, although any ordering is possible.
[0111] 6 illustrates steps of a channel identification method. As each channel is detected, e.g. after each step of the method, they are compared 210 with the channel index selected by the system settings, e.g. by the user. If any mismatch is detected, an alert 160 may be issued.
[0112] In some embodiments, the mismatch is automatically corrected, while in other embodiments, the mismatch is not corrected unless the user acknowledges it, for example after receiving a warning.
[0113] In some embodiments, the method further comprises calculating a confidence score for the outcome of any of the steps of the method, the confidence score being an indication of how reliable the outcome is.
[0114] This may be displayed to the user as part of a warning, allowing the user to make an informed decision as to whether the method's identification is more reliable than the current settings.
[0115] If the duration of the multi-channel audio signal is below a certain duration threshold, the confidence score may be multiplied by a weighting factor less than 1, such that durations below the duration threshold lead to a less reliable result.
[0116] A relatively long duration leads to more reliable results, since the weighting factor may be proportional to the duration divided by a duration threshold, which increases the accuracy of the weighting factor.
[0117] In one embodiment, if the duration is longer than the duration threshold, then the weighting factor is not applied or is equal to 1. This increases the accuracy of the weighting factor.
[0118] The weights may be calculated according to the following formula:
number
[0119] In most embodiments, more reliable results have a higher confidence score. The time duration threshold may be, for example, between 1 and 60 minutes, between 5 and 30 minutes, between 10 and 20 minutes, or a constant of 15 minutes. The duration threshold may instead be a relative length, such as 1 / 50, 1 / 20, 1 / 10, 1 / 5, 1 / 3, or 1 / 2 the length of the data.
[0120] The confidence score of the empty channel identification stage 110 may be proportional to the sound energy of the identified empty channel, whereby a relatively lower sound energy leads to a more reliable result.
[0121] In embodiments where channels with sound energy below an energy threshold may be identified as empty channels, the reliability of this identification depends on how far below the energy threshold the sound energy falls, with relatively lower sound energy leading to more reliable results.
[0122] Since the number of empty channels is unknown, a confidence score below a confidence threshold may cause the result of the empty channel identification stage 110 to be marked as unreliable, e.g., in short-term memory or as metadata, which may cause a warning to be displayed to the user and / or the empty channel identification stage 110 may be retried directly, e.g., if a mismatch is detected or if an incorrect number of LFE and / or center channels are identified.
[0123] The confidence score for the LFE channel discrimination stage 120 may be proportional to the difference between the subband sound energy in the low frequency region and the subband sound energy in all other frequency regions of the discriminated LFE channel, such that a relatively larger difference leads to a more reliable result.
[0124] The LFE channel should contain a substantially larger portion of the subband sound energy in the low frequency region compared to all other frequency regions, so a larger difference is more reliable.
[0125] The difference between the subband sound energies may be calculated by comparing the sums of the subband sound energies in different frequency regions.
[0126] The sum(s) may be further normalized to the size of each frequency region, respectively.
[0127] Alternatively, the difference between the subband sound energies may be calculated by comparing the average or normalized average of the subband sound energies in different frequency regions.
[0128] The normalized average is preferably normalized to the size of each frequency region.
[0129] This sum is preferred as it results in a larger difference and therefore a more standardized confidence score.
[0130] The low frequency region may be, for example, 400 Hz, 300 Hz, 200 Hz, 120 Hz, 100 Hz, or any sub-band below 50 Hz. The low frequency region may be determined based on the content of the audio signal.
[0131] In a further embodiment, the confidence score for the LFE channel discrimination stage 120 is proportional to the sum of the sub-band sound energy of the discriminated LFE channel in frequency regions higher than a frequency threshold, whereby a relatively lower sum leads to a more reliable result.
[0132] In this embodiment, content in the low frequency region is not used when determining the confidence score, which may be beneficial in some embodiments.
[0133] In one embodiment, the confidence score for the LFE channel discrimination stage 120 is: proportional to the difference between the subband sound energy in the low frequency region and the subband sound energy in all other frequency regions of the discriminated LFE channel, so that a relatively larger difference leads to a more reliable result, and proportional to the sum of the subband sound energy of the discriminated LFE channel in frequency regions above a frequency threshold, so that a relatively lower sum leads to a more reliable result.
[0134] In this embodiment, both of the indicators deemed most useful are used in combination, possibly with different weightings, to produce a reliable confidence score.
[0135] The frequency threshold may be, for example, 2000 Hz, 1000 Hz, 500 Hz, 400 Hz, 300 Hz, 200 Hz, 120 Hz, 100 Hz, or 50 Hz, or may be determined based on the content of the audio signal.
[0136] In some embodiments, the confidence score for the LFE channel discrimination stage 120 is proportional to the highest frequency signal present in the discriminated LFE channel, so that a relatively lower highest frequency signal leads to a more reliable result.
[0137] The presence or absence of an LFE channel may be determined based on an energy threshold, which may be adapted to ignore noise or may be so low that it is non-existent, so that the presence of any signal affects the confidence score.
[0138] In these embodiments, only the maximum frequency cutoff is used when determining the confidence score, which may be beneficial in some embodiments.
[0139] Since the presence of an LFE channel is unknown, a confidence score below a confidence threshold may cause the result of LFE channel determination stage 120 to be marked as unreliable, e.g., in short-term memory or as metadata, which may cause a warning to be displayed to the user and / or LFE channel determination stage 120 to be retried, e.g., if a mismatch is detected, or possibly even at a later stage directly if an incorrect number (e.g., multiple) of center channels and / or LFEs are identified.
[0140] The confidence score for the center channel identification stage 140 may be proportional to the independence and / or non-correlation of the identified center channel compared to the channels among the Y channels that are not identified as LFE channels, such that relatively higher independence and / or non-correlation leads to more reliable results.
[0141] The center channel should be independent and / or uncorrelated compared to the Y channels that are not identified as the LFE channel, and thus a high degree of independence and / or uncorrelation may be more reliable.
[0142] Where multiple calculation options for the confidence score at a stage of the method are available, they may be applied hierarchically.
[0143] The confidence score may be stored with the metadata.
[0144] Typically, a result with a confidence score below a confidence threshold (for any of the identification steps 110-150) may result in the channel identification method 100 being restarted, for example using longer data.
[0145] 7A-7B show a flow chart of method steps for channel identification. It shows a sequencing optimization of which checks and method steps are performed in which order to minimize computation. In this embodiment, a 5.1 surround sound file format is assumed, but with minor modifications, other formats are possible.
[0146] The first step is an empty channel identification step 110. The result of this step allows the method to reduce the number of possible configurations of the multi-channel audio signal to one or two options, which are listed after the result of the empty channel identification step 110.
[0147] The illustrated embodiment has six channels, but any other number is possible, adjusting for the consequences of the number of empty channels.
[0148] If the number of empty channels is five as a result of the empty channel identification step 110, the last channel is automatically identified as the center channel and is then output.
[0149] If the result of the empty channel identification step 110 is that the number of empty channels is 3, the identified empty channels are output and the remaining channels are assumed to be L, R, C. A channel pair splitting step 130 is used to find the pairs and the remaining channel is automatically identified as the center channel and then output along with the pair.
[0150] If the result of the empty channel identification stage 110 is that the number of empty channels is one, then the empty channel is double checked by using the LFE channel identification stage 120 to see if it has been mistaken for an LFE channel. If an LFE channel is found it is output, otherwise the empty channel is output. A channel pair splitting stage 130 is used to find two pairs out of the remaining five channels, and the remaining channel is automatically identified as the center channel and then output with its pair.
[0151] If the result of the empty channel identification stage 110 is that the number of empty channels is zero, then an LFE channel must be present if the input is formatted according to 5.1 surround sound. In an embodiment where a 7.1 format is possible, for example, the remaining six channels could be, for example, three pairs. The LFE channel is identified and output using an LFE channel identification stage 120. A channel pair splitting stage 130 is used to find two pairs out of the remaining five channels, and the remaining channel is automatically identified as the center channel and then output with the pair.
[0152] If the result of the empty channel identification stage 110 is that the number of empty channels is two, then the identified empty channel is output and the remaining channels can be either L, R, C, LFE or L, R, Ls, Rs. The LFE channel identification stage 120 is used next as it is relatively efficient. If an LFE channel is detected, then it is output and the remaining channels are L, R, C. Otherwise, the remaining channels are L, R, Ls, Rs. The channel pair splitting stage 130 is used to find one or two pairs out of the remaining three or four channels and the remaining channel, if any, is automatically identified as the center channel. Either way, the identified channel is then output.
[0153] If the number of empty channels as a result of the empty channel identification stage 110 is 4, the identified empty channel is output and the remaining channels can be either L, R or C, LFE. The LFE channel identification stage 120 is used next as it is relatively efficient. If the LFE channel is detected, the remaining channel is automatically identified as the center channel and then output together with the LFE channel. If the LFE channel is not detected, the remaining channel is an L, R pair. This pair may be output directly or the channel pair split stage 130 may be used as a precaution before the split pair is output.
[0154] If the result of the free channel identification step 110 is that the number of free channels is 6, then all channels are free. In that case, the free channels are output and the method ends.
[0155] The illustrated embodiment does not include a channel pair discrimination step 150, which would occur before the result of "Output L, R, C, (Ls, Rs)".
[0156] The illustrated embodiment does not include a center channel identification stage 140 other than identifying any single remaining channel as the center channel, however, one skilled in the art would be able to easily correct for it in accordance with the previously described embodiments. Furthermore, it is assumed that any single remaining channel is C rather than LFE, as this is more common, but in other embodiments where such an assumption is not made, the LFE channel discrimination stage 120 and / or center channel identification stage 140 may be performed.
[0157] Figure 8 shows the system architecture of the channel order detector 1. The channel order detector applies the method for channel identification according to the invention to detect the order of the channels.
[0158] The channel order detector 1 may be adapted to perform a method according to a computer program product. The computer program product comprises a non-transitory computer readable storage medium having instructions adapted to perform the method according to the invention when executed by a device having processing capabilities such as the channel order detector.
[0159] A multi-channel audio signal containing X>1 channels is input 801 to the channel order detector. The segment length 802 of the audio signal may be analyzed from the audio signal or may be input separately. The segment length 802 corresponds to the total length (in minutes) of the input data. Thus, if an audio file is input, the segment length 802 corresponds to the total length of the audio signal of that file.
[0160] The method for channel identification results in identified channels. The order detector may then use the identified channels to output an ordered sequence of labels for the channels 810.
[0161] Any number of confidence scores 820, as discussed above, may also be output relating to the reliability of the results of the method. The confidence scores may be normalized between 0 and 1, with a confidence score of 0 being unreliable and a confidence score of 1 being reliable, or vice versa.
[0162] The output sequence of detected labels may be used by a playback system to accurately match multiple channels to multiple sound sources, so that, for example, the center channel comes from the center speaker, etc.
[0163] A system including the channel order detector may further include a display. The method may include a display stage 160 in which the calculated confidence score(s) is / are displayed on the display 60.
[0164] The display 60 is beneficial in that it allows the user to receive feedback regarding the reliability of the method.
[0165] The display step 160 may further include displaying a warning if the calculated confidence score is below a confidence threshold.
[0166] This warning is beneficial in that it can alert the user to take action to improve the method, for example by stopping the method, rerunning the method, or by, for example, increasing the bit streaming rate and / or correcting the upstream fault.
[0167] The identified channel layout may be displayed in a display stage 160 (see FIG. 5), which may provide more meaningful feedback to the user.
[0168] In some embodiments, the display stage 160 further includes waiting for user input using a user interface, such as buttons or a touch screen, and thus the display 60 may include an interface for receiving such user input.
[0169] This prevents the method from continuing without the user having the possibility to analyze the results and provide feedback.
[0170] The identified channel layout may be approved by the user before being applied to the multi-channel audio signal, reducing the risk of a mistake being applied.
[0171] The user may not be prompted to accept an identified channel layout that is identical to the user's preferred layout. This scenario saves time and reduces user requirements since no changes are required to the playback system.
[0172] The display step 160 may further include displaying a warning if the identified channel layout differs from the user's preferred layout, as this may justify and / or force a change to the preferred layout and the user may want to know before this occurs.
[0173] The warning level may be proportional to the calculated confidence score(s). A confidence score indicating an unreliable result may, for example, justify a more easily perceptible warning, so that the user stops the method, restarts the method, and / or improves the method, or a less easily perceptible warning, so that the user ignores a warning that is likely to be false.
[0174] The display step 160 may further include allowing a user to manipulate the displayed data: the user may have information other than that available to the method, and may add to and / or modify the data available to the method.
[0175] The manipulated data may be used in the channel identification steps of the method, meaning that changes made during the execution of the method may be used to improve the channel identification steps as they are performed. The manipulated data may additionally or alternatively be used for subsequent executions of the method.
[0176] The display step 160 may further include allowing a user to select at least one segment of the signal to be ignored, thereby enabling the user to, for example, identify and remove imperfections in the audio signal that disrupt the method.
[0177] Fig. 9 illustrates steps of a channel identification method. The illustrated embodiment shows the different steps of the method being performed in different domains. In this embodiment, the empty channel identification stage 110, the LFE discrimination stage 120, the channel pair splitting stage 130 and the center channel identification stage 140 are performed in a time-frequency domain, such as the wavelet domain, and the channel pair discrimination stage 150 is performed in the spatial domain. This is achieved, for example, by transforming 910, 920 the multi-channel audio signal before certain stages in order to extract features in certain regions, and then inverse transforming after those stages have been performed.
[0178] This is just one possible embodiment; in other embodiments, different steps are performed in different areas than those shown, or, for example, the entire method is performed in one area.
[0179] The method 100 may further include applying 170 the identified channel layout to the multi-channel audio signal, which may include changing the order of the channels of the multi-channel audio signal, redirecting channels to the identified playback source, i.e., so that the left channel is output by the left speaker, or any other physical and / or digital manipulation of the multi-channel audio signal to conform to the identified layout that is the result of the method for channel identification.
[0180] In some embodiments, the identified channel layout is applied only if the calculated confidence score(s) exceeds a confidence threshold.
[0181] If the identified channel layout is not reliable, applying the identified channel layout may worsen the radiated sound image, so a confidence threshold may be used to prevent this.
[0182] The applying stage 170 may include using any present metadata to apply the identified channel layout to the multi-channel audio signal, which may make the applying stage 170 more effective and may be used by any further systems in the broadcast chain.
[0183] The channel layout identified by this method may be applied in real-time to the multi-channel audio signal as it is streamed to a speaker system.
[0184] The proposed method is very computationally efficient and can be applied in real-time without significant delay to the playback.
[0185] Initial results may be inaccurate, resulting in a low confidence score, but as more data is collected as the audio signal is played, the score increases.
[0186] A real-time embodiment of the method may include: an initialization that clears all data buffers and obtains channel numbers. After any new data is acquired, channel identification may be performed on all available data. Features of previous data may be used to keep the complexity consumption low. Inconsistent data may be accepted. If no decision can be made for certain channels based on the available data, they may be labeled as unknown and have a confidence score of 0. Initially, the confidence scores of all channels are low due to a global weighting factor. After enough data is received, the identification remains constant and the confidence scores may fluctuate slightly.
[0187] A multichannel audio signal may be a multichannel surround sound file or stream for content creation, analysis, conversion and playback systems, which are strongly influenced by the channel layout.
[0188] At least one of the steps of the method may use a machine learning based method, which may be a decision tree, Adaboost, GMM, SVM, HMM, DNN, CNN and / or RNN.
[0189] Machine learning may be used to further improve the efficiency and / or reliability of the method.
[0190] SVM for channel pair detection may be taken as an example. As shown in Equation 3, the inter-channel spectral distance between channels i and j in frame l is denoted by D. i,j (l). Then, the total frequency band is divided into 1, 2, …, or K different bands, and the inter-channel spectral distance is calculated, and the average inter-channel spectral distance
number
number
[0191] Figure 10 shows a flow chart of the channel pair splitting stage 130. The channel pair detection is usually performed on non-empty and non-LFE channels to be more efficient. Channel pairs may be detected when the number of unknown channels is 2 or more.
[0192] Matching symmetrical channels in the channel pair splitting stage 130 may further include comparing temporal, spectral, auditory and / or other domain features to calculate sound energy distribution and variance between the audio energy of each channel and matching the most symmetrical channels as pairs. Symmetrical channels are found as audio channels with substantially similar and / or symmetrical audio signal content by analyzing sound energy distribution and variance. Symmetrical audio signal content may include, for example, similar background sounds and different foreground sounds, similar bass sounds and different descant sounds, respectively, or vice versa. Symmetrical audio content may further include synchronous sounds, such as different parts of a single chord, or sounds starting in one channel and ending in another channel.
[0193] If the features of two channels are very close but quite different from the features of the other channel, or if the correlation between the two channels is higher than the other channel, then those two channels may be split into a channel pair.
[0194] Symmetric channel matching in the channel pair splitting step 130 further includes calculating 1010 an inter-channel spectral distance between the channels using the calculated sound energy distribution and variance of each channel in the short-term, medium-term and / or long-term duration, where the inter-channel spectral distance is a normalized pairwise measure of the distance between two matching sound energy sub-bands in each channel, summed over multiple sub-bands; and matching the channels with the shortest distance as a pair.
[0195] The distance measures used may be Euclidean distance, Manhattan distance and / or Minkowski distance.
[0196] All the following examples are in the frequency domain, but other domains are possible. In addition to the embodiments using time-frequency features, features derived from signal transformation or other methods of signal analysis theory may also be used, for example, to perform pair detection and / or confidence score estimation. In addition to heuristic rule-based methods as described above, machine learning-based methods such as regression, decision trees, Adaboost, GMM, HMM or DNN may also be used, for example, for pair detection and / or confidence score estimation.
[0197] In one embodiment, the distance between channels i and j in frame l is
number
[0198] The average of the calculated inter-channel spectral distances over time may be calculated and used to pair-match channels with the smallest average distance, which is used to measure the long-term similarity between channels.
[0199] In one embodiment, the average inter-channel distance between the channels is:
number
[0200] Instead of or in addition to the average distance, the minimum and / or maximum inter-channel distance may be used, however the average is preferred since although paired channels are similar on average, they are not necessarily similar all the time, e.g., in every frame.
[0201] In embodiments using inter-channel spectral distance, center channel identification stage 140 may further include analyzing the calculated inter-channel spectral distances for any remaining unpaired channels of the Y channels that have not been identified as an LFE channel or split into pairs to identify the center channel, which further increases the accuracy of center channel identification stage 140.
[0202] The confidence score for the center channel identification stage 140 may be proportional to the calculated inter-channel spectral distance between the identified center channel and other channels among the Y channels that are not identified as LFE channels, such that a relatively symmetric distance leads to more reliable results.
[0203] The center channel preferably has a symmetrical distance to the other channel that is not identified as the LFE channel, i.e., the paired channel, so that the relatively symmetrical distance leads to more reliable results.
[0204] The confidence score for the center channel identification stage 140 may be directly proportional to the confidence score of the channel pair splitting stage 130, if present.
[0205] For example, if the center channel identification stage 140 involves simply identifying the remaining channels, then the reliability of the center channel identification stage 140 is directly proportional to the reliability of the channel pair splitting stage 130. In other embodiments, the reliability of the pair matching may also affect which available channels should be identified as center channels, and thus may directly affect the reliability of the center channel identification stage 140.
[0206] Matching symmetric channels in the channel pair splitting stage 130 may further include comparing the correlation of the sound energy distribution of each channel and matching the most highly correlated channels as pairs, which is a simple and efficient computation but only works in some embodiments.
[0207] The correlation measures used may be cosine similarity, Pearson correlation coefficient and / or correlation matrix.
[0208] The channel pair splitting stage 130 may further include, for each of the Y channels that is not identified as the LFE channel, measuring and / or importing from previous measurements, if any, at least one parameter used for the channel pair matching calculations.
[0209] The indicator may for example be the sound energy measured in the empty channel identification stage 110 or the LFE channel discrimination stage 120. This improves the efficiency of the method 100.
[0210] If channel pairs are matched differently according to the feature(s) used to match them, a hierarchy of the feature(s) may be used to determine which pairing to apply.
[0211] The hierarchy may be, for example, such that some types of metrics are preferred over others, for example average inter-channel spectral distance is preferred over maximum inter-channel spectral distance or correlation of sound energy distributions.
[0212] Channel pair splitting stage 130 may continue pairing any unpaired channels among the Y channels that are not identified as the LFE channel until fewer than two channels remain.
[0213] There may be more than two pairs of channels, such as the front and rear pairs in the 5.1 audio format, so if there are more than two channels remaining, there are likely to be further channel pairs among them, and further pairs can be split off.
[0214] The channel pair splitting stage 130 may further include assigning the first received channel of the multi-channel audio signal in each pair as the left channel and the last listed channel in each pair as the right channel.
[0215] In multi-channel audio signals, it is conventional to list the left channel of each pair before the right channel, so the method is more efficient by assuming that this is always the case.
[0216] The division of channels into pairs and / or the assignment of left and right channels, if any, may be stored using metadata.
[0217] The confidence score for the channel pair splitting stage 130 may be proportional to the symmetry index of the matched pair(s), such that a relatively high symmetry index leads to more reliable results.
[0218] Correctly matched pairs preferably have a high degree of symmetry, so if the result of the channel pair splitting stage 130 has pairs with a relatively high degree of symmetry, it is relatively reliable.
[0219] The confidence score for the channel pair splitting stage 130 may be proportional to the calculated inter-channel spectral distance between the matched pair(s), such that a relatively smaller distance leads to a more reliable result.
[0220] Correctly matched pairs preferably have a short distance between each other, so if the result of the channel pair splitting step 140 has pairs with a relatively short distance, it is relatively reliable.
[0221] The confidence score for the channel pair splitting stage 140 may be proportional to the calculated inter-channel spectral distance between each channel in the matched pair(s) and any other channel among the Y channels that is not identified as an LFE channel or is not a matched channel, such that a relatively larger distance leads to a more reliable result.
[0222] Correctly matched pairs preferably have a long distance to the other channel, so if the result of the channel pair splitting step 140 has pairs with a relatively long distance to the other channel, it is relatively reliable.
[0223] At least a portion of the channel pair splitting step may be redone 1040 with a different sub-band split when calculating the inter-channel spectral distance if the confidence score for this step is below a confidence threshold 1030.
[0224] By varying the subband partitioning, more reliable results may be achieved, and in some embodiments, the subband partitioning is varied 1030 until a satisfactory reliability of the channel pair partitioning stage 140 is achieved, for example through a confidence threshold or a pair score threshold.
[0225] The pair score is an index for comparing the likelihood that members of a pair can be grouped into another pair. The pair score threshold is a pre-determined threshold for the pair score. If the pair score is higher than the pair score threshold, the result of the channel pair splitting step 140 is sufficiently reliable.
[0226] A version of this is shown in the flow chart of Figure 10. First, the average inter-channel spectral distance is calculated for all possible pairs. Then, for the pair with the lowest inter-channel spectral distance, a pair score is calculated 1020. If the pair score is not high enough to make a decision, a different time-frequency segmentation is used to obtain a new average inter-channel spectral distance and corresponding pair score. Trials can be performed until all channels are paired or some termination condition is met. If more than two channels are still undetected, their confidence scores are all set to 0.
[0227] The confidence score may be further weighted by a global weighting factor to take into account the total length of the data. Channel pair detection is performed on all unknown channels until only one channel remains.
[0228] The pair score may be used as a confidence score or as part of a confidence score.
[0229] In one embodiment, the pair score for a pair of channels i and j is calculated as follows:
number
[0230] The pair score can be calculated for any possible pair, or only for the two channels with the lowest average inter-channel spectral distance, i.e., channels i,j in the above formula. The pair score is a measure of the confidence in splitting them as a channel pair.
[0231] The pair score compares the inter-channel spectral-channel distance between a candidate channel pair i,j with each of the other channels to ensure that the two channels are similar to each other while being distinct from any of the other channels. If there are also other channels similar to channel i or j, P i,j is much lower than 1, and therefore implies low reliability.
[0232] 11 shows a flow chart of the channel pair position discrimination step 150. The channel pair discrimination step 150 involves discriminating between the paired channels as forward pairs, side pairs, rear pairs, and / or any other positional pairs.
[0233] A channel pair discrimination step 150 is part of the method for channel identification, preferably performed after the pair splitting step 130 .
[0234] Many multichannel audio signals contain multiple channel pairs, such as 5.1, which includes a front pair and a rear pair, and it is therefore beneficial for a method for channel identification to be able to distinguish between position pairs and correctly identify position pairs as such.
[0235] The directional stability of the front sound image is typically maintained for the majority of its duration, and the rear channels typically carry information that can enhance the overall sound image.
[0236] The channel pair discrimination step 150 may include a step of calculating 1120 an inter-pair level difference for each pair, where the inter-pair level difference is proportional to the decibel difference in the sum of the subband sound energies of each pair, and the pair with the highest relative level is discriminated as the front pair.
[0237] Alternatively or additionally, amplitude panning may be performed in conjunction with calculating the level difference between the pairs. Amplitude panning involves generating virtual sound sources.
[0238] Most virtual sound sources may be generated to appear from the front, which results in the front pair having a relatively higher amplitude than other positional pairs, and thus the pair with the highest amplitude may be identified as the front pair.
[0239] The panning method may further include moving the rear pair out of phase, so that the relatively out of phase pair may be identified as the rear peta.
[0240] Since the front pair should be closest to the center channel, the front pair is traditionally the pair with the highest relative level 1140 .
[0241] In one embodiment, the pairwise level difference between a pair of channels i, j and another pair of channels m, n, both in band b, is calculated for each time-frequency tile according to the following formula:
number
[0242] The inter-pair level difference between the pairs is not always high enough, since differences less than 2 dB may not carry useful information, so a segment of the signal may be selected whose content would result in a larger inter-pair level difference between the pairs.
[0243] Thus, the channel pair discrimination step 150 may further include the steps of selecting, for each channel in each pair, one or more segments of the signal for which the subband sound energy of the signal exceeds an energy threshold; and calculating the level difference between the channel pair using only these segments.
[0244] By selecting segments that have a large amount of information in the form of subband sound energy above an energy threshold, the level difference between the pairs can be increased.
[0245] The channel pair discrimination stage 150 may further comprise a stage 1150 of selecting one or more segments of the signal for each pair for which the absolute pair-wise level difference exceeds an absolute threshold; and calculating the channel pair-wise level difference using only these segments.
[0246] Selecting segments with a high threshold can increase the average pairwise level difference. Many multichannel audio signals have similar output in multiple channels during parts of the signal. These parts do not contribute to the pairwise level difference and can therefore be safely ignored.
[0247] As a complement to measuring the absolute pairwise level difference, the average pairwise level difference in a segment that is relatively small compared to the total length of the signal may also be used, or this may be used instead. If the selection of the segment does not give a sufficiently high average pairwise level difference, selection with a higher absolute threshold may achieve it.
[0248] Thus, if the relatively highest average inter-pair level difference falls below the level threshold (as determined in step 1130), the step of calculating the inter-pair level difference of channels may be repeated using a higher absolute threshold 1150 until the average inter-pair level difference is sufficiently high.
[0249] Alternatively or additionally, the pair with the highest relative combined directional coherence with the identified center channel may be identified as the front pair if the highest relative average inter-pair level difference is below a level threshold.
[0250] In one embodiment, segment selection is abandoned and directional consistency with the identified central channel is used instead to distinguish pairs: the pair with the closest orientation to the central channel is also closest to the central channel.
[0251] Directional coherence is a measure of the similarity of two channels in the time domain, which is related to the sound image direction, which implies the phase difference between the channels.
[0252] The directional difference can be used to measure the consistency of the direction of the main sound source between the two channels. A simplified measure of directional consistency according to one embodiment is as follows:
number
[0253] Anterior pairs should traditionally have relatively higher directional coherence with each other than other location pairs, and posterior pairs should traditionally have relatively lower directional coherence with each other than other location pairs.
[0254] The signals in the forward pair are typically time-aligned to represent directional sources, and therefore have high correlation and low delay. This means that there are more identical components in the forward pair compared to the rear pair. The directional difference measures this, as shown in Equation 7. If the signals in channels i and j are identical, this means that they are in phase and X=1, otherwise X<1. If the two channels are out of phase, X=0.
[0255] In another embodiment, if the relatively highest average inter-pair level difference is below the level threshold and the absolute threshold is above the maximum threshold 1160, the pair with the relatively highest combined directional consistency with the identified center channel 1170 is distinguished as the forward pair 1180.
[0256] This embodiment is shown in Figure 11. In this embodiment, the entire signal is first selected 1110, but the average inter-pair level difference does not reach a high enough level to exceed the level threshold, and selection of a segment did not result in a high enough average inter-pair level difference, so instead directional coherence with the identified center channel is used to distinguish the pairs.
[0257] The reason for the failure of segment selection is that the average pairwise level difference did not reach a high enough level to exceed the level threshold, and the absolute threshold was so high that the segments above it were not long enough to allow for the computation of pairwise level differences.
[0258] The level threshold may be a constant between 2 and 3 dB. The maximum threshold for the absolute threshold may be 2 dB, and / or any threshold where the total length of the selected segment is less than, for example, 20% of the non-silence signal length, or less than, for example, 1 minute.
[0259] The maximum absolute threshold is relevant when the selected segment or segments of the signal for each channel in each pair, whose average inter-channel spectral distance exceeds the distance threshold, are not long enough to calculate the inter-channel level difference: if the total length of the selected segments is less than 20% of the non-silent signal length or less than 1 minute, the useful signal is too short.
[0260] The distinction between the location pairs may be based on their similarity to the identified center channel, where the pair most similar to the identified center channel may be distinguished as a forward pair and the pair least similar to the identified center channel may be distinguished as a backward pair.
[0261] The center channel is conventionally in front of the sound image, so the front pair should resemble it more than, say, the rear pair.
[0262] The similarity to the identified center channel can be based on time-frequency features, spatial features, sound image direction, inter-channel phase difference, and / or inter-channel pair-level difference.
[0263] Additionally or alternatively, similarity to the identified center channels may be calculated using delay panning, and the pair with the highest directional consistency with the center channel is identified as the front pair.
[0264] The time-frequency features are examined first, followed by the spatial features, since amplitude panning is used most frequently and the calculation of the time-frequency features requires less time.
[0265] A channel directional pattern may be generated to compare the distance of the pair from the center of the channel pair, and then the channel pair closer to the center channel is detected as the forward pair.
[0266] Where different pairs are distinguished as same location pairs depending on the features used to make the distinction, those features may be prioritized according to a hierarchy.
[0267] The hierarchy may depend, for example, on the confidence scores, the metrics used, or the thresholds used.
[0268] The distinction between channel pairs may be stored using metadata.
[0269] A confidence score may be calculated on the results of the channel pair discrimination stage 150 .
[0270] The confidence score of the channel pair discrimination stage 150 may be proportional to the calculated inter-channel spectral distance between the identified center channel and a paired channel among the Y channels that is not identified as the LFE channel, such that a relatively small inter-channel spectral distance between the front pair and the center channel results in a more reliable result.
[0271] The pair that is closest to the identified center channel is classified as the forward pair, and the pair that is least similar to the identified center channel is classified as the backward pair. This index reflects this.
[0272] The confidence score for the channel pair discrimination stage 150 may be proportional to the directionality of the channels of the split pair, such that a relatively large difference between the directionality results in a more reliable result.
[0273] The pair with the direction closer to the central channel is also closer to the central channel and is therefore a forward pair. Thus, a larger difference leads to a more reliable discrimination. Absolute differences and / or ratios of different pairs can be used.
[0274] For similar reasons, the confidence score for the channel pair discrimination stage 150 may be proportional to the directionality of the identified central channel and the channels of the split pair, so that relatively small differences between the directionality of the central channel and one of the pairs lead to more reliable results.
[0275] The confidence score for the channel pair discrimination stage 150 may be proportional to the calculated inter-pair level difference of the paired channels, such that a relatively higher average level difference leads to more reliable results.
[0276] An average pair-to-pair level difference of more than 2 dB gives useful information, and the higher it is, the more useful information there is. More information leads to more reliable results.
[0277] The confidence score for the channel pair discrimination stage 150 may be directly proportional to the confidence scores of the channel pair splitting stage 130 and / or the center channel identification stage 140, if present.
[0278] The channel pair discrimination stage 150 is unreliable if the channel pair splitting stage 130 is unreliable. Furthermore, many possible confidence score calculations for the channel pair discrimination stage 150 depend on the central channel identification stage 140. Thus, to save computations, pre-computed confidence scores for the channel pair splitting stage 130 and / or the central channel identification stage 140 may be reused.
[0279] The confidence score for the channel pair discrimination stage 150 may be proportional to the length of the selected segment or segments of the signal, such that a relatively longer segment or segments leads to more reliable results.
[0280] A short length of the selected segment would make the calculation of pairwise level differences unreliable. The absolute length of the selected segment and / or the ratio of the length of the selected segment to the total length of the data may be used.
[0281] At least a portion of the channel pair discrimination step 150 may be re-run with a different data segment if the confidence score for that step falls below a confidence threshold.
[0282] This ensures that the results of the channel pair discrimination stage 150 are reliable.
[0283] Further embodiments of the present disclosure will be apparent to those skilled in the art after reviewing the above description. Although the present specification and drawings disclose embodiments and examples, the present disclosure is not limited to these specific examples. Numerous modifications and variations can be made without departing from the scope of the present disclosure defined by the appended claims. Any reference signs appearing in the claims shall not be understood as limiting the scope thereof.
[0284] Furthermore, variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the present disclosure, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.
[0285] The above-described systems and methods may be implemented as software, firmware, hardware, or a combination thereof. For example, aspects of the present application may be embodied, at least in part, in an apparatus, a system including multiple apparatuses, a method, a computer program product, etc. In a hardware implementation, the division of tasks among the functional units described above does not necessarily correspond to a division into physical units. Conversely, one physical component may have multiple functions, and one task may be performed by multiple cooperating physical components. Some or all of the components may be implemented as software executed by a digital signal processor or microprocessor, or as hardware, or as an application specific integrated circuit. Such software may be distributed on a computer readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term "computer storage media" includes both volatile and non-volatile, removable and non-removable media, implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Additionally, communication media typically embodies computer-readable instructions, data structures, program modules or other data in the form of a modulated data signal, such as a carrier wave or other transport mechanism, and is well known to those skilled in the art to include any information delivery media.
[0286] Various aspects of the present invention can be understood from the enumerated example embodiments (EEE) listed below: [EEE1] 1. A method for channel identification of a multi-channel audio signal comprising X>1 channels, the method comprising: identifying (110) any empty channels among the X channels, resulting in a subset of Y≦X non-empty channels; determining (120) whether a low frequency effects (LFE) channel is present among the Y channels, and if an LFE channel is present, identifying the determined channel among the Y channels as an LFE channel; splitting remaining channels of the Y channels that are not identified as LFE channels into any number of channel pairs by matching symmetric channels; and identifying (140) any remaining unpaired channels of the Y channels that have not been identified as an LFE channel or split into pairs as center channels. method. [EEE2] The method of EEE1, further comprising the step of differentiating (150) the channels divided into pairs among forward pairs, side pairs, rear pairs, and / or any other position pairs. [EEE3] The method according to EEE2, wherein the channel pair discrimination step includes calculating an inter-pair level difference between the pairs; the inter-pair level difference is proportional to the decibel difference of the sum of the subband sound energy of each pair, and the pair having the relatively higher level is discriminated as the forward pair. [EEE4] The method according to claim 8, wherein the channel pair discrimination step further comprises amplitude panning in conjunction with the calculation of the level difference between the pair, the amplitude panning comprising generating a virtual sound source. [EEE5] The method of any one of claims 3 to 4, wherein the channel pair discrimination step further comprises selecting one or more segments of the signal for each pair in which the subband sound energy of the signal exceeds an energy threshold; and calculating the pair-to-pair level difference of the pair using only these segments. [EEE6] A method as described in any one of EEE3 to 5, wherein the channel pair discrimination step further comprises selecting one or more segments of the signal in each pair in which the absolute pair-to-pair level difference exceeds an absolute threshold; and calculating the pair-to-pair level difference using only these segments. [EEE7] The method according to EEE6, wherein if the relatively highest average pairwise level difference falls below the level threshold, the step of calculating the pairwise level differences of channels is repeated using a higher absolute threshold. [EEE8] The method of any one of EEE3 to 7, wherein the pair having the highest combined directional consistency with the identified center channel is distinguished as the front pair if the highest average inter-pair level difference is below a level threshold. [EEE9] The method according to EEE7, wherein the pair with the highest relative combined directional coherence with the identified central channel is distinguished as a forward pair if the highest relative average inter-pair level difference is below a level threshold and the absolute threshold is above a maximum threshold. [EEE10] The method according to EEE9, wherein the maximum absolute threshold is 2 dB. [EEE11] The method according to any one of EEE8 to 10, wherein the directional consistency is a measure of similarity of two channels in the time domain and is related to the sound image direction, which implies a phase difference between the channels. [EEE12] The method according to any one of EEE7 to 11, wherein the level threshold is a constant between 2 and 3 dB. [EEE13] The method of any one of EEE2-12, wherein the distinction between pairs of locations is based on the similarity of the pairs of locations to the identified central channel. [EEE14] The method according to EEE13, wherein a pair that is most similar to the identified center channel is classified as the forward pair and a pair that is least similar to the identified center channel is classified as the rear pair. [EEE15] The method according to any one of claims 8 to 10, wherein the similarity to the identified centre channel is based on time-frequency features, spatial features, sound image direction, phase difference between the channels and / or level difference between pairs. [EEE16] The method of any one of EEE13 to 15, wherein the similarity to the identified center channel is calculated using a delay pan, and the pair with the highest directional consistency with the center channel is distinguished as the front pair. [EEE17] The method according to any one of claims 8 to 12, wherein the similarity to the identified central channel is calculated by generating directional patterns of the channels and comparing the distance of the channel pairs from the center to the pair, and the pair closer to the central channel is distinguished as the forward pair. [EEE18] The method according to any one of EEE2 to 17, wherein, depending on the features used to make the distinction, if different pairs are distinguished as identical location pairs, the features are prioritized according to a hierarchy. [EEE19] The method according to any one of claims 8 to 10, wherein said distinction between pairs of channels is stored using metadata. [EEE20] 20. The method of any one of EEE1 to 19, wherein the empty channel identifying step further comprises the step of measuring sound energy in each of the X channels. [EEE21] The method of EEE20, wherein the sound energy in each of the X channels is measured at short, medium and / or long durations. [EEE22] 22. The method according to claim 20, wherein a channel is identified as empty if its total sound energy is below an energy threshold. [EEE23] 3. The method of any one of claims 20 to 22, wherein a channel is identified as empty if each of its subband sound energies is below an energy threshold. [EEE24] 10. The method of any one of claims 8 to 10, wherein the sound energy is measured in the temporal, spectral, wavelet and / or auditory domains. [EEE25] The method of any one of EEE1 to 25, wherein the identification of empty channels is stored using metadata. [EEE26] The method according to any one of EEE20 to 25, wherein the LFE channel determination step further comprises a step of determining whether an LFE channel is present using the measured sound energy in each of the Y channels. [EEE27] The method according to any one of EEE1-26, wherein the LFE channel determination step further comprises the step of measuring frequency bands in which sound energy above an energy threshold is present in each of the Y channels. [EEE28] The method according to EEE27, wherein the frequency bands in which sound energy above an energy threshold is present in each of the Y channels are measured in short, medium and / or long durations. [EEE29] A method according to any one of EEE26 to 28, wherein an LFE channel is determined to be present among the Y channels if the sum of subband sound energy in a low frequency region of a channel is significantly greater than the sum of subband sound energy in all other frequency regions of that channel. [EEE30] The method according to EEE29, wherein the sum of the subband sound energies in each frequency region is further normalized by the size of each frequency region. [EEE31] The method of any of claims 8 to 10, wherein any such channel is identified as the LFE channel. [EEE32] 2. The method according to claim 1, wherein the low frequency region includes a subband below 200 Hz. [EEE33] The method according to any one of EEE26 to EEE32, wherein an LFE channel is determined to be present among the Y channels if a channel contains subband sound energy above an energy threshold only in a frequency region below a frequency threshold. [EEE34] The method according to EEE33, wherein only such any channel is identified as the LFE channel. [EEE35] The method of any one of claims 33 to 34, wherein the frequency threshold is 200 Hz or greater. [EEE36] The method of any one of EEE26 to 35, wherein if it is determined that there are multiple LFE channels among the Y channels, then only one is identified as the LFE channel according to a hierarchy of the feature(s) used to determine whether an LFE channel is present. [EEE37] The method of any one of EEE1-37, wherein the identity of the LFE channel is stored using metadata. [EEE38] The method of any one of EEE1 to EEE38, wherein the matching of symmetrical channels in the channel pair splitting step further comprises matching the most symmetrical channels as pairs by comparing temporal, spectral, auditory and / or other domain features to calculate sound energy distribution and variance between the audio signals of each channel. [EEE39] 8. The method according to claim 6, wherein the matching of symmetric channels in the channel pair splitting step further comprises the steps of calculating an inter-channel spectral distance between the channels using sound energy distribution and variance in short-term, medium-term and / or long-term duration of each channel, the inter-channel spectral distance being a normalized pairwise measure of the distance between two matching sound energy sub-bands in each channel summed over multiple sub-bands; and matching the channels with the shortest distance as a pair. [EEE40] The method according to EEE39, where the distance measures used are Euclidean distance, Manhattan distance and / or Minkowski distance. [EEE41] The method of any one of EEE38 or 40, wherein the average over time of the calculated inter-channel spectral distances is calculated and used to pair-match channels that have the shortest average distance to each other. [EEE42] 8. The method according to any one of claims 39 to 41, wherein the step of identifying the centre channel further comprises the step of analysing a calculated inter-channel spectral distance of any remaining unpaired channels of the Y channels that have not been identified as an LFE channel or split into pairs to identify the centre channel. [EEE43] A method according to any one of EEE1 to 42, wherein the matching of symmetrical channels in the channel pair splitting step further comprises comparing the correlation of the sound energy distribution of each channel and matching the most correlated channels as a pair. [EEE44] The method according to EEE43, wherein the correlation measures used are cosine similarity, Pearson correlation coefficient, and / or correlation matrix. [EEE45] The method according to any one of EEE38 to 44, wherein the channel pair splitting step further comprises, for each channel among the Y channels that is not identified as an LFE channel, measuring, or importing from previous measurements, if any, at least one parameter used for the calculations of matching the channels in pairs. [EEE46] A method according to any one of EEE38 to 45, wherein if the channel pairs are matched differently depending on the feature or features used to match the pairs, a hierarchy of the features used determines which pairing applies. [EEE47] 56. The method of claim 45, wherein the channel pair splitting step continues pairing any unpaired channels among the Y channels that are not identified as an LFE channel until fewer than two channels remain. [EEE48] The method of any one of EEE1 to 47, wherein the channel pair splitting step further comprises assigning a first received channel of the multi-channel audio signal in each pair as the left channel and assigning a last listed channel in each pair as the right channel. [EEE49] A method according to any one of EEE1 to 48, wherein the division into pairs of channels and / or the allocation of left and right channels, if any, is stored using metadata. [EEE50] The method of any one of EEE1 to 49, wherein the center channel identification step further comprises calculating the independence and / or uncorrelation of the remaining unpaired channels of the Y channels that have not been identified as an LFE channel or split into pairs compared to other channels of the Y channels, and identifying the center channel as the most independent and / or uncorrelated channel. [EEE51] The method of EEE50, wherein the calculation of the independence and / or decorrelation of any remaining unpaired channels among the Y channels that are not identified as LFE channels or split into pairs is calculated only in comparison to channels that are split into pairs. [EEE52] The method according to claim 50 or 51, dependent on at least one of EEE2 to 19, wherein the center channel identification step is performed after the channel pair discrimination step, and the calculation of independence and / or decorrelation of any remaining unpaired channels among the Y channels that have not been identified as an LFE channel or split into pairs is calculated only in comparison with the channel discriminated as the forward pair. [EEE53] The method of any one of EEE1 to EEE52, wherein the identification of the center channel is stored using metadata. [EEE54] The method of any one of EEE1 to 53, further comprising the step of calculating a confidence score for the result of any of the method steps, said confidence score being an indication of how trustworthy that result is. [EEE55] The method according to claim 54, wherein if the duration of the multi-channel audio signal is below a certain duration threshold, the confidence score is multiplied by a weighting factor less than one, whereby durations below the duration threshold lead to a less reliable result. [EEE56] The method according to EEE55, wherein the weighting factor is proportional to the duration divided by the duration threshold, such that a relatively longer duration leads to a more reliable result. [EEE57] The method according to any one of claims 55 to 56, wherein if said duration is longer than said duration threshold then said weighting factor is not applied or is equal to 1. [EEE58] 8. The method of any one of claims 55 to 57, wherein the duration threshold is a constant between 5 and 30 minutes. [EEE59] The method according to any one of claims 54 to 58, wherein the confidence score for the empty channel identification stage is proportional to the sound energy of the identified empty channel, a relatively lower sound energy leading to a more reliable result. [EEE60] The method according to any one of EEE54 to 59, wherein a confidence score for the LFE channel determination stage is proportional to the difference between the subband sound energy in the low frequency region and the subband sound energy in all other frequency regions of the determined LFE channel, with a relatively larger difference leading to a more reliable result. [EEE61] The method of EEE60, wherein the subband sound energy difference is calculated by comparing sums of the subband sound energies in the different frequency regions. [EEE62] The method of any one of claims 60 to 61, wherein the low frequency region includes any subband below 200 Hz. [EEE63] The method according to any one of EEE54 to 62, wherein the confidence score for the LFE channel determination stage is proportional to the sum of the sub-band sound energy of the determined LFE channel in frequency regions higher than a frequency threshold, such that a relatively lower sum leads to a more reliable result. [EEE64] The method according to EEE63, wherein the frequency threshold is greater than or equal to 200 Hz. [EEE65] 8. The method according to any one of claims 54 to 64, wherein the confidence score for the LFE channel determination stage is proportional to the highest frequency signal present in the determined LFE channel, such that a relatively lower highest frequency signal leads to a more reliable result. [EEE66] The method according to any one of claims 54 to 65, wherein the confidence score for the channel pair splitting stage is proportional to a symmetry index of the matched pair(s), such that a relatively high symmetry index leads to a more reliable result. [EEE67] The method according to any one of claims 54 to 66, wherein the confidence score for the channel pair splitting stage is proportional to the calculated inter-channel spectral distance between the matched pair(s), such that a relatively smaller distance leads to a more reliable result. [EEE68] 8. The method according to any one of claims 54 to 67, wherein the confidence score for the channel pair splitting stage is proportional to the calculated inter-channel spectral distance between each channel in the matched pair(s) and any other channel among the Y channels that is not identified as the LFE channel or is not a matched channel, with a relatively larger distance leading to a more reliable result. [EEE69] A method according to any one of EEE66 to 68, wherein at least a part of the channel pair splitting steps is re-performed using a different subband split when calculating inter-channel spectral distances if the confidence score for that step is below a confidence threshold. [EEE70] 8. The method according to any one of claims 54 to 69, wherein the confidence score for the centre channel identification stage is proportional to the independence and / or non-correlation of the identified centre channel compared to a channel among the Y channels that is not identified as an LFE channel, such that a relatively higher independence and / or non-correlation leads to a more reliable result. [EEE71] 8. The method according to any one of claims 54 to 70, wherein the confidence score for the centre channel identification stage is proportional to the calculated inter-channel spectral distance between the identified centre channel and other channels of the Y channels that are not identified as LFE channels, with a relatively symmetrical distance leading to a more reliable result. [EEE72] The method according to any one of EEE54 to 71, wherein the confidence score for the central channel identification stage is directly proportional to the confidence score of the channel pair splitting stage, if a channel pair splitting stage is present. [EEE73] A method according to any one of EEE54-72 dependent on at least one of EEE2-19, wherein a confidence score is calculated for the result of the channel pair discrimination step. [EEE74] 8. The method according to claim 6, wherein the confidence score for the channel pair discrimination stage is proportional to a calculated inter-channel spectral distance between the identified central channel and a paired channel among the Y channels that is not identified as an LFE channel, and wherein a relatively smaller inter-channel spectral distance between the front pair and the central channel leads to a more reliable result. [EEE75] The method according to any one of claims 8 to 10, wherein the confidence score for the channel pair discrimination step is proportional to the directionality of the channels of a split pair, with a relatively larger difference between the directionality leading to a more reliable result. [EEE76] The method according to any one of claims 83 to 85, wherein the confidence score for the channel pair discrimination stage is proportional to the directionality of the identified central channel and the channels of the split pairs, and a relatively small difference in directionality between the central channel and one of the pairs leads to a more reliable result. [EEE77] The method according to any one of claims 83 to 86, wherein the confidence score for the channel pair discrimination step is proportional to the calculated pairwise level differences of the channel pairs, with a relatively higher average level difference leading to a more reliable result. [EEE78] The method according to any one of EEE73 to EEE77, wherein the confidence score for the channel pair discrimination stage is directly proportional to the confidence scores of the channel pair splitting stage and / or the central channel identification stage, if they exist. [EEE79] The method according to any one of EEE73 to EEE78, dependent at least on EEE4 or EEE5, wherein a confidence score for the channel pair discrimination stage is proportional to a length of the selected segment or segments of the signal, with a relatively longer segment or segments leading to a more reliable result. [EEE80] The method of any one of EEE73 to EEE70, wherein at least a portion of the channel pair discrimination step is re-performed on a different data segment if the confidence score for that step falls below a confidence threshold. [EEE81] A method according to any of EEE54 to 80, wherein when multiple calculation options for the confidence score for a stage of the method are available, they are applied hierarchically. [EEE82] The method according to any one of EEE54 to 81, wherein the confidence scores are stored using metadata. [EEE83] The method according to any one of claims 54 to 82, further comprising a display step (160) in which the calculated confidence score is displayed on a display (60). [EEE84] The method according to EEE83, wherein the displaying step further comprises displaying a warning if the calculated confidence score is below a confidence threshold. [EEE85] A method according to any one of claims 84 to 87, further comprising a display step in which the identified channel layout is displayed. [EEE86] The method of any one of EEE83 to 85, wherein the displaying step further comprises waiting for user input using a user interface such as a button or a touch screen. [EEE87] The method according to any of claims 85 to 86, wherein the identified channel layout is approved by a user before being applied to the multi-channel audio signal. [EEE88] The method according to EEE87, wherein the user is not prompted to accept an identified channel layout that is identical to the user's preferred layout. [EEE89] The method of any one of EEE83 to 88, wherein the displaying step further comprises the step of displaying a warning if the identified channel layout differs from a user-preferred layout. [EEE90] The method according to claim EEE89 dependent on any one of claims EEE54 to 82, wherein the alert level is proportional to the calculated confidence score(s). [EEE91] 8. The method according to any one of claims 83 to 890, wherein the displaying step further comprises allowing a user to manipulate the displayed data. [EEE92] The method according to claim 8, wherein the manipulated data is used in the channel identification steps of the method. [EEE93] The method of any one of EEE83 to 92, wherein the displaying step further comprises allowing a user to select at least one segment of the signal to be ignored. [EEE94] The method of any one of EEE1 to EEE93, further comprising the step of applying (170) the identified channel layout to the multi-channel audio signal. [EEE95] The method according to claim EEE94 dependent on any one of claims EEE54 to EEE82, wherein the identified channel layout is applied only if the calculated confidence score(s) exceeds a confidence threshold. [EEE96] 8. The method according to claim 6, wherein the applying step comprises using any present metadata to apply the identified channel layout to the multi-channel audio signal. [EEE97] The method of any one of EEE1 to EEE96, wherein the channel layout identified by the method is applied in real-time to the multi-channel audio signal as it is streamed to a speaker system. [EEE98] The method of any one of EEE1 to EEE97, wherein the multi-channel audio signal is a multi-channel surround sound file or stream for content creation, analysis, conversion and playback systems. [EEE99] The method of any one of EEE1-98, wherein at least one step of the method uses a machine learning based method. [EEE100] The method according to EEE99, wherein the machine learning based method is a decision tree, Adaboost, GMM, SVM, HMM, DNN, CNN and / or RNN. [EEE101] 100. An apparatus configured to identify channels of a multi-channel audio signal, the apparatus (1) comprising a circuit configured to perform a method (100) according to any one of claims 1 to 100. [EEE102] A computer program product comprising a non-transitory computer readable storage medium having instructions adapted to perform the method of any one of EEE1 to 100 when executed by a device (1) having processing capability.
Claims
1. 1. A method for channel identification of a multi-channel audio signal comprising X>1 channels, the method comprising: identifying 110 empty channels, if any, among said X channels, resulting in a subset of Y≦X non-empty channels; determining (120) whether a low frequency effects (LFE) channel is present among the Y channels, and if an LFE channel is present, identifying the determined channel among the Y channels as an LFE channel; splitting (130) any remaining channels of the Y channels that are not identified as LFE channels into any number of channel pairs by matching symmetric channels; and identifying (140) any remaining unpaired channels of the Y channels that have not been identified as an LFE channel or split into pairs as center channels. method.
2. 2. The method of claim 1, further comprising: differentiating (150) the channels divided into pairs among front pairs, side pairs, rear pairs, and / or any other position pairs, wherein the channel pair differentiating step includes calculating an inter-pair level difference between every two pairs; the inter-pair level difference being proportional to the decibel difference of the sum of the subband sound energies of each pair, and the pair with the highest relative level being differentiated as the front pair.
3. the channel pair discrimination step further comprises: selecting one or more segments of the signal for each channel of each pair in which an absolute pair-wise level difference exceeds an absolute threshold; and calculating the pair-wise level difference of channels using only those segments. The method of claim 2.
4. The method described in claim 3, wherein if the relatively highest average pair-wise level difference falls below the level threshold, the step of calculating the channel pair-wise level difference is repeated using a higher absolute threshold.
5. 5. The method of claim 3 or 4, wherein if the relatively highest average inter-pair level difference is below a level threshold and the absolute threshold is above a maximum threshold, the pair with the relatively highest directional coherence is classified as a front pair, where the directional coherence is a measure of similarity of two channels in the time domain, which is related to sound image direction, which implies a phase difference between channels.
6. 6. The method of claim 1, wherein the empty channel identification step further comprises measuring the sound energy in each of the X channels, and if the total sound energy of a channel is below an energy threshold, the channel is identified as empty.
7. 7. The method of claim 1, wherein an LFE channel is determined to be present among the Y channels if the sum of the subband sound energy in a low frequency region of a channel that is any subband below 200 Hz is significantly greater than the sum of the subband sound energy in all other frequency regions of the channel.
8. 8. The method of claim 1, wherein the symmetric channel matching in the channel pair splitting step further comprises: calculating an inter-channel spectral distance between the channels using the calculated sound energy distribution and variance of each channel, the inter-channel spectral distance being a normalized pairwise measure of the distance between two matching sound energy sub-bands in each channel, summed over multiple sub-bands; and matching the channels with the shortest distance as a pair.
9. 9. The method of claim 1, wherein the channel pair splitting step continues pairing any unpaired channels of the Y channels that are not identified as LFE channels until fewer than two channels remain.
10. 10. The method according to claim 1, further comprising the step of calculating a confidence score for any of the results of the method steps, the confidence score being an indication of how reliable that result is, and wherein if the duration of the multi-channel audio signal is below a certain duration threshold, the confidence score is multiplied by a weighting factor less than 1, whereby durations below the duration threshold lead to a less reliable result.
11. The method of claim 10, further comprising a display step in which the calculated confidence score is displayed on a display, and a warning is displayed if the calculated confidence score is below a confidence threshold and / or if the identified channel layout differs from the user's preferred layout.
12. The method of any one of claims 1 to 11, further comprising the step of applying the identified channel layout to the multi-channel audio signal.
13. 13. The method of claim 1, wherein the channel layout identified by the method is applied in real-time to the multi-channel audio signal as it is streamed to a speaker system.
14. 14. The method according to claim 1, wherein at least one of the method steps uses a machine learning based method, the machine learning based method being a Decision Tree, Adaboost, GMM, SVM, HMM, DNN, CNN and / or RNN.
15. 15. An apparatus configured to identify channels of a multi-channel audio signal, the apparatus comprising circuitry configured to perform the method of any one of claims 1 to 14.
16. A computer program product having a non-transitory computer readable storage medium having instructions adapted to perform the method of any one of claims 1 to 14 when executed by a device having processing capability.
Citation Information
Patent Citations
Audio signal encoding method, encoding device, decoding method, and decoding device
JP2011066868A
System and method for adaptive audio signal generation, coding, and rendering
JP2014522155A
Detection of audio channel configuration
US20120195433A1
