Channel identification of multi-channel audio signals

Abstract

A method for channel identification of a multi-channel audio signal including X>1 channels is provided. The method comprises the following steps: identifying any empty channels among the X channels, thereby generating a subset having Y≤X non-empty channels; determining whether there is a low frequency effect (LFE) channel among the Y channels, and when it is determined that there is an LFE channel, identifying the determined channel among the Y channels as the LFE channel; dividing the remaining channels among the Y channels that are not identified as the LFE channels into any number of channel pairs by matching symmetric channels; and identifying any remaining unpaired channels among the Y channels that are not identified as the LFE channels or divided into pairs as center channels.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to PCT Patent Application No. PCT / CN2019 / 103813 filed on August 30, 2019, U.S. Provisional Patent Application No. 62 / 912,279 filed on October 8, 2019, and European Patent Application No. 19204516.9 filed on October 22, 2019, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to the field of channel identification, and in particular to a channel identification method, device and software for a surround sound system. Background Art

[0004] Audio signals usually go through several conversions before reaching a multichannel system. During these conversions, channels may be swapped or corrupted. Surround sound processes usually do not include functions for channel identification, abnormal channel detection, or channel swap detection, and usually use default layout settings. If the channel layout of the incoming sound data does not match the settings at the time of processing, the channels are swapped.

[0005] The current standard saves the swapped channel indexes as metadata in the surround data, which makes the metadata unreliable and harmful to subsequent processes. If the surround contains some abnormal channels, the error may not be detected, so the error may be passed to the next process.

[0006] Therefore, improvements are needed in this regard. Summary of the invention

[0007] In view of the above, it is therefore an object of the present invention to overcome or alleviate at least some of the problems discussed above. In particular, it is an object of the present disclosure to provide channel layout identification of an audio signal based on channels, rather than channel layout identification based on metadata added by a sound codec. This feature can make the identification independent of the encoding format or number of channels and unaffected by mismatched metadata. Spatial auditory impressions are important for multi-channel surround sound and are typically generated by panning sound sources through mixing. The channel identification method described herein extracts spatial information to restore the channel layout. Further and / or alternative objects of the present invention will be clear to readers of the present disclosure.

[0008] According to a first aspect of the present invention, there is provided a method for channel identification of a multi-channel audio signal including X>1 channels, the method comprising the following steps: identifying any empty channel among the X channels, thereby generating a subset having Y≤X non-empty channels; determining whether there is a low frequency effect (LFE) channel among the Y channels, and when it is determined that there is an LFE channel, identifying the determined channel among the Y channels as the LFE channel; dividing the remaining channels among the Y channels that are not identified as the LFE channels into any number of channel pairs by matching symmetric channels; and identifying any remaining unpaired channels among the Y channels that are not identified as the LFE channels or divided into pairs as center channels.

[0009] In the context of this specification, the term "channel identification" should be understood as when the channels of an audio signal are swapped and / or damaged, the channel identification can be used to find the correct settings of the audio signal to restore the audio signal to its original intent. The term "channel identification" includes functions such as abnormal channel detection and / or channel swap detection.

[0010] In the context of this specification, the term "multi-channel audio signal" should be understood as an audio signal having at least two audio channels. An audio channel is a sequence of sound signals, preferably different from at least another channel of the multi-channel audio signal. The audio signal may be in the format of, for example, an audio file, an audio clip or an audio stream.

[0011] In the context of this specification, the term "empty channel" should be understood as an audio channel whose sound signal content is below a certain threshold. The threshold may be, for example, a total energy content threshold or an average energy content threshold.

[0012] In the context of this specification, the term "low frequency effects (LFE) channel" should be understood as an audio channel whose sound signal content substantially, mainly or only comprises energy below a frequency threshold (such as 200 Hz).

[0013] In the context of this specification, the term "symmetrical channels" should be understood as audio channels having sufficiently similar and / or symmetrical sound signal content. Symmetrical sound signal content may, for example, include similar background sounds and different foreground sounds, similar fundamental tones (e.g., low frequencies) and different treble tones (e.g., high frequencies), respectively, or vice versa. Symmetrical sound signal content may further include synchronized sounds, such as different parts of a single chord or a sound that starts in one channel and ends in another channel.

[0014] In the context of this specification, the term "center channel" should be understood as an audio channel that is substantially independent of the other channels, which includes the most general content of the other audio channels. The present disclosure focuses on embodiments with only one center channel, which is the current standard for multi-channel audio signals, however if the current standard develops further, the method according to the first aspect may be adapted accordingly.

[0015] The inventors have realised that identification of the centre channel is more difficult than many of the other steps. Therefore, computing power can be saved by performing the centre channel identification step as the last step in the channel identification method, thereby reducing the computation to finding the remaining channel after all other channels have been identified and optionally verifying it as the centre channel.

[0016] Similar efficiencies related to sequencing (ie, the specific order of steps of the channel identification method described herein) will be discussed with respect to specific embodiments, however many of them are generally applicable to most embodiments.

[0017] Besides saving computational power, the ranking can be further used to improve the reliability of the methods by starting with the most reliable ones.

[0018] In a preferred embodiment, sorting can be used both to save computing power and to increase the reliability of the method.

[0019] According to some embodiments, the method further includes a step of distinguishing channels divided into pairs between front pairs, side pairs, rear pairs and / or any other position pairs, wherein the channel pair distinguishing step includes calculating the inter-pair level difference between every two pairs; the inter-pair level difference is proportional to the decibel difference of the sum of the sub-band sound energy of each pair; wherein the pair with the relatively highest level is distinguished as the front pair.

[0020] Many multi-channel audio signals include more than one channel pair; such as 5.1, which includes a front pair and a rear pair. Therefore, a method for channel identification that can distinguish between position pairs and correctly identify the position pairs is beneficial. Inter-pair level difference is an effective and accurate measure for distinguishing between position pairs.

[0021] According to some embodiments, the channel pair distinguishing step further comprises selecting one or more segments of the signal for each channel in each pair where the absolute inter-pair level difference is above an absolute threshold; and using only these segments to calculate the inter-pair level difference for the pair, wherein if the relatively highest average inter-pair level difference is below the level threshold, the step of calculating the inter-pair level difference for the pair is repeated with a higher absolute threshold.

[0022] The level differences between pairs are not always high enough, as differences below, for example, 2 dB may not be informative. Therefore, it is beneficial to select segments of the signal with content that is likely to produce larger level differences between pairs. If the selection of segments does not result in a high enough average inter-pair level difference, selection with a higher absolute threshold may achieve this.

[0023] In these embodiments the absolute pairwise level differences are checked point by point, so the selected segment may contain some isolated frames.

[0024] In other embodiments, the absolute values ​​are examined segment by segment, the maximum absolute pairwise level difference is compared to an absolute threshold or the average absolute pairwise level difference is compared to an absolute threshold. This results in the selected segments being quantified by the length of the segment examined.

[0025] According to some embodiments, if the relatively highest average pair-to-pair level difference is below a level threshold and the absolute threshold is above a maximum threshold, the pair with the relatively highest directional consistency is distinguished as the front pair, wherein the directional consistency is a measure of the similarity of two channels in the time domain, the directional consistency being related to the sound image direction, which in turn means the phase difference between the channels.

[0026] In these embodiments, the selection of segments failed to produce a sufficiently high average inter-pair level difference. Therefore, directional consistency is used instead to distinguish these pairs. The pairs with the highest directional consistency are distinguished as the front pairs. The signals in the front pairs are usually time-aligned to represent directional sound sources, so they have higher correlation and lower delay, and thus have higher directional consistency. This means that there are more identical components in the front pairs compared to the rear pairs.

[0027] Selection of segments failed because the highest average pairwise level difference did not reach a high enough level to exceed the level threshold, and the absolute threshold was so high that the segments exceeding the absolute threshold were not long enough to calculate the pairwise level difference. A useful signal may be considered too short if the total length of the selected segments is shorter than, for example, 20% (or any other defined percentage) of the length of the non-silent signal or shorter than, for example, 1 minute (or any other defined length).

[0028] Directional coherence measures the proportion of identical components in a signal by comparing sample values ​​at different points in the time domain. Higher similarity between the signals in two channels means higher correlation and lower latency. Paired channels usually have correlated signals, and the signals in the leading pair are usually time-aligned to represent directional sound sources.

[0029] Alternatively, the directional consistency of the combination with the identified center channel may be used to distinguish the pairs. The pair that is closest in direction to the center channel is also closest to the center channel (ie the pair identified as the front pair).

[0030] According to some embodiments, the empty channel identification step further comprises measuring the acoustic energy in each of the X channels, wherein the channel is identified as empty if the total acoustic energy of the channel is below an energy threshold.

[0031] The acoustic energy is usually measured using subbands for each channel by summing the amplitude of each frequency in each subband. This produces an efficient way to identify empty channels, even though there may be noise in the empty channels due to coding or other reasons.

[0032] The energy threshold may be, for example, -80 dB to -60 dB, preferably -70 dB. As an alternative or in addition to measuring the total sound energy, the average sound energy in a time segment may also be measured, wherein the time segment may be between 1 second and 10 seconds.

[0033] Empty channels can be the result of, for example, unusual equipment, stereo commercial breaks during multi-channel television programs, and multi-channel surround sound upmixed from original stereo or mono sound.

[0034] According to some embodiments, if the sum of subband sound energies in the low frequency region of the channel, i.e., any subband below 200 Hz, is significantly higher than the sum of subband sound energies in all other frequency regions in the channel, it is determined that an LFE channel exists in the Y channels.

[0035] This is beneficial because the LFE channel is less likely to be missed. 200 Hz is the cutoff frequency in the low frequency region, intended to ensure that the LFE channel is not missed while also reducing false positives. Typically, the threshold is 120 Hz, but the threshold may preferably be set to a higher value because the normal channels transmit signals in a wider frequency band.

[0036] According to some embodiments, the matching of symmetrical channels in the channel pair division step further comprises calculating an inter-channel spectral distance between the channels using the calculated energy distribution and variance of each channel; the inter-channel spectral distance is a normalized pairwise measure of the distance between two matching energy subbands in each channel summed over a plurality of subbands; and matching the channels into pairs having the shortest distances from each other.

[0037] The inter-channel spectral distance is a simple and accurate measure of symmetry. The mathematical distance is a similarity measure that can be weighted in various ways. The distance measure used can be the Euclidean distance, the Manhattan distance and / or the Minkowski distance.

[0038] According to some embodiments, the channel pairing step continues pairing any unpaired channels of the Y channels that are not identified as the LFE channel until fewer than two channels remain.

[0039] There may be more than two pairs of channels, such as a front pair and a rear pair. Therefore, if there are more than two channels remaining, there may be more channel pairs among them, and there may be more pairs to divide.

[0040] According to some embodiments, the channel pair division step further comprises assigning a first received channel of the multi-channel audio signal in each pair as a left channel and assigning a last listed channel in each pair as a right channel.

[0041] In multi-channel audio signals, it is customary to list the left channel before the right channel in each pair, so this method is more efficient by assuming that this is always the case.

[0042] According to some embodiments, the method further comprises calculating a confidence score for any result of the steps of the method, the confidence score being a measure of the reliability of the result, wherein if the duration of the multi-channel audio signal is below a certain duration threshold, the confidence score is multiplied by a weighting factor less than one, such that a duration less than the duration threshold produces a less reliable result.

[0043] In order to diagnose errors or measure improvements, it may be useful to know how reliable each result of the steps of the method is. If the duration of the multi-channel audio signal is too short, the identification made is unreliable because too little data can be used in the calculation. Therefore, weighting factors can be used.

[0044] According to some embodiments, the method further comprises a display step, wherein the calculated confidence score is displayed on a display; and wherein a warning is displayed if the calculated confidence score is below a confidence threshold and / or if the identified channel layout is different from a user's set layout.

[0045] The display is beneficial because the user can receive feedback about the reliability of the method. This allows the user to make an informed decision about whether the method's identification is more reliable than the current settings. The warning is beneficial because it can prompt the user to take action, such as to stop the method, redo the method, or improve the method by, for example, increasing the bitstream rate and / or fixing upstream interference. If the identified channel layout is different from the user's setup layout, the settings and / or the identified channel layout may be incorrect, which may require action, such as by the device or the user.

[0046] According to some embodiments, the method further comprises the step of applying the identified channel layout to the multi-channel audio signal.

[0047] The applying step may include: changing the channel order of the multi-channel audio signal; redirecting the channels to the identified playback sources, i.e., so that the left channel is output by the left speaker; or any other physical and / or digital manipulation of the multi-channel audio signal to conform to the identified layout as a result of the channel identification method.

[0048] According to some embodiments, the channel layout identified by the method is applied to the multi-channel audio signal in real-time when the multi-channel audio signal is streamed to a speaker system.

[0049] Since the proposed method is computationally very efficient, it can be applied in real time without causing any significant delay in the playback.

[0050] The first result may be inaccurate and have a low confidence score, and then as the audio signal plays and more data is acquired, the confidence score increases.

[0051] According to some embodiments, at least one of the steps of the method uses a machine learning based method, wherein the machine learning based method is a decision tree, Adaboost, GMM, SVM, HMM, DNN, CNN and / or RNN.

[0052] Machine learning may be used to further improve the efficiency and / or reliability of the method.

[0053] According to a second aspect of the present invention, there is provided an apparatus configured for identifying channels of a multi-channel audio signal, the apparatus comprising circuitry configured to perform the method according to the first aspect of the present invention.

[0054] According to a third aspect of the invention, there is provided a computer program product comprising a non-transitory computer readable storage medium having instructions adapted to, when executed by a device having processing capabilities, perform a method according to the first aspect of the invention.

[0055] The second and third aspects may generally have the same features and advantages as the first aspect.

[0056] It should further be noted that, unless explicitly stated otherwise, the invention relates to all possible combinations of features. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above as well as further objects, features and advantages of the present invention will be better understood through the following illustrative and non-limiting detailed description of preferred embodiments of the present invention with reference to the accompanying drawings, in which like reference numerals will be used for similar elements, and in which:

[0058] Figure 1showing a menu of surround sound in different formats according to some embodiments,

[0059] Figure 2 shows the channel layout of a 5.1 surround sound system according to some embodiments,

[0060] Figure 3 shows a flow chart of a sound broadcast chain according to some embodiments,

[0061] Figure 4 a diagram showing steps of a method for channel identification according to some embodiments,

[0062] Figure 5 a diagram showing steps of a method for channel identification according to some embodiments,

[0063] Figure 6 a diagram showing steps of a method for channel identification according to some embodiments,

[0064] FIG. 7A to FIG. 7B A flowchart showing the steps of a method for channel identification according to some embodiments,

[0065] Figure 8 shows a system architecture of a channel order detector according to some embodiments,

[0066] Fig. 9 a diagram showing steps of a method for channel identification according to some embodiments,

[0067] Fig.10 A flowchart showing the channel pair division steps according to some embodiments, and

[0068] Fig.11 A flow chart showing the steps of channel pair position differentiation according to some embodiments. DETAILED DESCRIPTION

[0069] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the invention are shown.The systems and apparatus disclosed herein will be described during operation.

[0070] The present disclosure generally relates to the problem of swapped or damaged channels of a multi-channel audio signal. In order to restore the channels to their intended state, the inventors have discovered that channel identification can be used. In the following, the multi-channel audio signal is a 5.1 audio signal. However, this is only by way of example, and the methods and systems described herein can be used for channel identification of any multi-channel audio signal (e.g., 7.1).

[0071] Figure 1Schematically shows the menu of a workstation for multi-channel sound processing. This is an example of different formats of the widely used 5.1 channels.

[0072] Current standard practice involves simply selecting a default format and if the channel layout of the input sound data does not match the settings at processing time, the channels will be swapped. The swapped channel indices can be saved as metadata to the surround sound data so that the channels are always swapped correctly. However, if future systems use different defaults, the metadata will become unreliable and harmful to future processes.

[0073] If the multi-channel audio signal further includes a corrupted channel, the current standard does not detect this anomaly and thus the error will propagate to future systems.

[0074] Figure 2 A typical layout of a 5.1 surround sound system is shown. If the content of any speaker of the system is swapped or any channel is damaged or emptied, the audio experienced by the listener is different from the original intent. For example, if the front R speaker content and the surround R speaker content are swapped, the symmetry of the speaker pair is destroyed, or if the front L speaker content is empty, an important part of the overall sound image may be lost. The sound image in the original surround sound data cannot be reproduced, and the spatial impression is confused and becomes annoying to the listener.

[0075] Unusual channel(s) may be detected because their index or entire layout may look unusual. Any swapped channels can also be found by comparing the detected channel layout with the channel layout in the user's settings.

[0076] Throughout this disclosure, the terms "surround pair" and "rear pair" will be used interchangeably in order to generalize the disclosure for further possible positional pairs, such as in a 7.1 surround sound system where the surround pairs are replaced by side and rear pairs.

[0077] Figure 3 An example of a high-level sound system for a typical broadcast chain is shown. This example shows the surround sound data flow in a typical broadcast chain and implies that the surround sound is converted several times during a typical workflow before playback. Figure 1 As discussed, errors in metadata may propagate through such a workflow. Further, channels may be swapped or corrupted at each step of the workflow.

[0078] The process starts with production, which includes channel-based content, object-based content and / or scene-based content that constitutes advanced sound file formats. Advanced sound file formats are output by production and input into distribution.

[0079] Distribution includes adapting the Advanced Audio File Format distribution to the Advanced Audio Format. The Advanced Audio Format is exported by the distribution and imported into the broadcast.

[0080] Broadcasts include a bifurcation between high bandwidth broadcasts and low bandwidth broadcasts. Broadcasts for low bandwidth broadcasts render advanced sound formats into legacy stream formats. Legacy stream formats are exported by broadcasts and imported into low bandwidth connections / legacy broadcasts.

[0081] Low bandwidth connections / legacy broadcasts include direct reproduction to legacy devices.

[0082] The broadcast of the high bandwidth broadcast adapts the advanced sound format to the broadcast stream format. The broadcast stream format is output by the broadcast and input to the high bandwidth connection / broadcast.

[0083] High bandwidth connections / broadcasts include devices that render as speaker layouts or binaural layouts for Hi-Fi, TV, phones, tablets, etc.

[0084] Since metadata is unreliable, the inventors have discovered a channel identification method that relies solely on the audio content of a multi-channel audio signal to detect abnormal channels. The detector can detect the layout of the channels based on all available data, and can further provide a confidence score for the estimated channel index to indicate reliability. (Multiple) abnormal channels may be detected because their index or the entire layout may look abnormal. By comparing the detected channel layout with the channel layout in the user's settings, any channel swaps can also be found.

[0085] Typically, the audio data includes: a front sound image from a center channel and possibly a pair of front channels, where directional stability is maintained for most of the duration; left and right channels that carry information for balanced sound, and the channels may be processed in pairs; and rear channels that carry information that may enhance the overall sound image. The audio data may further include a separate low-frequency channel to complete the sound image with low frequencies. If multi-channel surround sound accompanies a video or image, the sound image preferably coincides with the visual image and the designed listening area.

[0086] By basing the channel identification on the audio data, the identification is independent of the encoding format or number of channels and is not affected by mismatched metadata. The spatial auditory impression is important for multi-channel surround sound and is usually generated by panning the sound sources through mixing. Channel identification extracts the spatial information to restore the channel layout.

[0087] Figure 4 A schematic diagram of an embodiment of a channel layout identification method 100 is shown. The method 100 comprises five steps, which are performed in a specific order in order to minimize the required calculations.

[0088] The method 100 starts with a multi-channel audio signal comprising X>1 unidentified channels. The first step is an empty channel identification step 110, since this is the least computationally demanding step.

[0089] The empty channel identification step 110 includes measuring the acoustic energy in each of the X channels in order to identify any empty channels, thereby producing a subset having Y≤X non-empty channels.

[0090] The acoustic energy in each of the X channels may be measured in short, medium, and / or long durations and may be measured in the time domain, spectral domain, wavelet domain, and / or auditory domain.

[0091] Depending on the content of the channel, different terminology may be useful.

[0092] The time domain includes information about the sound pressure values ​​at different points in time. The spectral domain includes frequency information in spectral components, achieved by transforming the content of the sound channels. The wavelet domain includes time and frequency information in a wavelet multiresolution decomposition, achieved by transforming the content of the sound channels. The auditory domain is the normal, untransformed domain that includes information about the auditory nerve response caused by hearing the signal.

[0093] The auditory domain can be used for channel identification. For example, an auditory filter-based decomposition, such as a mel / bark filter bank, can be used in each method step. In such an embodiment, the specific loudness of each critical band is used instead of the subband energy in equation 1.

[0094] Wavelet transforms are also suitable for signal decomposition and can provide time-frequency features for subsequent method steps.

[0095] A channel is identified as empty if the following conditions are met: its total sound energy is below an energy threshold; or each of its sub-band sound energy is below an energy threshold. A sub-band is a range of energy.

[0096] One definition of subband energy is:

[0097]

[0098] in, is the subband energy of channel c in frequency band b in frame l (l = 1..L), L is the total number of frames, X c (k, l) is the spectral amplitude of frequency index k in frame l of channel c, and f l 、f h are respectively the lowest and highest index of the frequency bins of frequency band b.

[0099] This definition is measured over a short period of time. For a time block of one or a few frames, calculate E b，cIf both the mean and the standard deviation of all time blocks are below a certain threshold, then subband b of channel c is detected as null.

[0100] Alternatives include spectrum-related measures such as bandpass filtered signals and auditory rate maps.

[0101] Metadata may be used to store identification of empty channels.

[0102] The LFE determination step 120 follows and includes determining whether a low frequency effects (LFE) channel exists among the Y channels, and when it is determined that the LFE channel exists, identifying the determined channel among the Y channels as the LFE channel.

[0103] The LFE determination step 120 may further include determining whether an LFE channel exists using the acoustic energy in each of the Y channels measured in the empty channel identification step 110. This saves computational effort.

[0104] The LFE determination step 120 may further include measuring frequency bands in each of the Y channels where acoustic energy above the energy threshold exists. This does not require measuring acoustic energy in the empty channel identification step 110.

[0105] The frequency bands in each of the Y channels where acoustic energy above the energy threshold exists may be measured over a short, medium, and / or long duration.

[0106] Determining the presence of an LFE channel in the Y channels may include checking whether the sum of the subband energies in the low frequency region of the channel is significantly higher than the sum of the subband energies in all other frequency regions in the channel. This is beneficial because the LFE channel is less likely to be missed.

[0107] As an alternative to summing the sub-band acoustic energies, for example an average value and / or a maximum value may be used.

[0108] Any such channel may be identified as an LFE channel. The low frequency region may be, for example, any subband below 400 Hz, 300 Hz, 200 Hz, 120 Hz, 100 Hz or 50 Hz. The low frequency region may be determined based on the content of the audio signal.

[0109] In fact, depending on the embodiment, any frequency between 200 Hz and 2000 Hz may belong to the low frequency region or the high frequency region. Therefore, the low frequency region may be determined based on the specific embodiment. Alternatively, it may be beneficial to only look at the sub-bands below 200 Hz and above 2000 Hz.

[0110] The highest frequency of a signal may depend on the sampling rate of the signal. Therefore, it may be beneficial to only look at sub-bands between 2000Hz and half the sampling rate.

[0111] Determining the presence of an LFE channel in the Y channels may include checking whether the channel includes only sub-band acoustic energies above an energy threshold in a frequency region below a frequency threshold. This is beneficial because any channel other than the LFE channel may not be detected, however, if, for example, it contains noise or has a different low frequency region than expected, the LFE channel may not be detected. In some embodiments, only any such channel is identified as an LFE channel.

[0112] The frequency threshold may be, for example, 2000 Hz, 1000 Hz, 500 Hz, 400 Hz, 300 Hz, 200 Hz, 120 Hz, 100 Hz, or 50 Hz, or may be determined based on the content of the audio signal.

[0113] If it is determined that several LFE channels exist among the Y channels, only one LFE channel may be identified as the LFE channel according to the level of the feature(s) used to determine whether the LFE channel exists.

[0114] Since most multi-channel audio signals have only one LFE channel at most, a level can be used to determine which of several possible LFE channels is identified as the LFE channel. The level can, for example, include a hard threshold or a maximum difference in sub-band energy between the low frequency region and other frequency regions.

[0115] Metadata may be used to store the identified LFE channel.

[0116] The next step is channel pair division step 130 and includes dividing the remaining channels of the Y channels not identified as LFE channels into any number of channel pairs by matching symmetrical channels. Fig.10 The channel pair division step 130 is discussed further.

[0117] A center channel identification step 140 follows and includes identifying any remaining unpaired channels of the Y channels that are not identified as LFE channels or divided into pairs as center channels.

[0118] The center channel identification step 140 may further include calculating the independence and / or uncorrelation of any remaining unpaired channels of the Y channels that are not identified as LFE channels or divided into pairs compared to other channels of the Y channels, and identifying the center channel as the most independent and / or uncorrelated channel.

[0119] This may be calculated, for example, based on measuring the content of the different channels in, for example, the time domain, the spectral domain, the wavelet domain and / or the auditory domain.

[0120] The calculation of independence and / or uncorrelation of any remaining unpaired channels in the Y channels that are not identified as LFE channels or divided into pairs may be calculated only compared to the channels divided into pairs. This is because the center channel is typically the most independent and / or least correlated with paired channels.

[0121] In another embodiment, the center channel identification step 140 occurs after the channel pair distinction step 150, and the calculation of independence and / or irrelevance is calculated only in comparison to the channels that are distinguished as the front pair.

[0122] This is because the center channel is typically the least independent and / or least uncorrelated with the front pair of channels, yet still independent and / or uncorrelated. Therefore, if independence and / or uncorrelation is found, the identification of the center channel is highly reliable because the probability of a false positive is reduced. Comparing the center channel to all pairs would be more reliable, but would consume more resources.

[0123] Either of these embodiments are beneficial because they are highly reliable; however, these embodiments may require a large amount of computation. Therefore, in an advantageous embodiment, any remaining channel is identified as a center channel without verification.

[0124] If there are more than one channel remaining, all channels may be identified as the center channel, or an error may be assumed and the channel identification method restarted. All steps may be re-executed or only steps determined to be likely to be erroneous may be executed.

[0125] If an even number of channels remain, the repeated steps may be, for example, always the empty channel identification step 110 and / or the LFE channel determination step 120, since these even number of channels may result in different parities, and if an odd number of channels different from one remain, the repeated steps may be, for example, always the channel pair division step 130 and / or the channel pair distinguishing step 150, since these odd number of channels will result in the same parity.

[0126] Repeated steps may additionally or alternatively be associated with a confidence score for the step, which would be combined with Figure 6 Further explanation.

[0127] Metadata may be used to store the identification of the center channel.

[0128] Figure 5 The steps of the method for channel identification are shown. This embodiment further includes a display step 160 and an application step 170, which are respectively combined with Figures 8 to 9 Further discussion. Since efficiency is achieved by reusing previous results, Figure 5 The sequence shown in is the preferred order, however any sequence is possible.

[0129] Figure 6 The steps of the method for channel identification are shown. As each channel is detected, for example after each step of the method, they are compared 210 with the settings of the system (eg the channel index selected by the user). If any mismatch is detected, a warning 160 may be issued.

[0130] In one embodiment, the mismatch is automatically fixed. In another embodiment, the mismatch is not fixed unless the user confirms it, for example, after receiving an alert.

[0131] In some embodiments, the method further comprises calculating a confidence score for any result of the steps of the method, the confidence score being a measure of the reliability of the result.

[0132] This may be displayed to the user as part of a warning to allow the user to make an informed decision as to whether the method's identification is more reliable than the current setting.

[0133] If the duration of the multi-channel audio signal is below a certain duration threshold, the confidence score may be multiplied by a weighting factor that is less than one, such that durations less than the duration threshold produce less reliable results.

[0134] The weight factor may be proportional to the duration divided by the duration threshold, such that relatively longer durations produce more reliable results. This increases the accuracy of the weight factor.

[0135] In one embodiment, if the duration is longer than the duration threshold, no weighting factor is applied or the weighting factor is equal to 1. This increases the accuracy of the weighting factor.

[0136] The weight can be calculated according to the following equation:

[0137]

[0138] Where L is the length of the data on which the channel identification is based, and L thd is the duration threshold. This means that if the data is below the duration threshold, the identification is unreliable.

[0139] In most embodiments, relatively more reliable results have relatively higher confidence scores.

[0140] The duration threshold may be a constant, such as 1-60 minutes, 5-30 minutes, 10-20 minutes or 15 minutes. The duration threshold may alternatively be a relative length, such as one-fiftieth, one-twentieth, one-tenth, one-fifth, one-third or one-half of the data length.

[0141] The confidence score of the empty channel identification step 110 may be proportional to the acoustic energy of the identified empty channel, such that relatively lower acoustic energies produce more reliable results.

[0142] In embodiments where channels with acoustic energy below an energy threshold can be identified as empty channels, the reliability of such identification will depend on the extent to which the acoustic energy is below the energy threshold. Therefore, relatively low acoustic energy produces more reliable results.

[0143] Since the number of empty channels is unknown, a confidence score below the confidence threshold may result in the result of the empty channel identification step 110 being marked as unreliable, for example in short-term memory or as metadata. If a mismatch is detected, or if an incorrect number of LFE channels and / or center channels are identified, this may result in displaying a warning to the user and / or directly re-performing the empty channel identification step 110, for example.

[0144] The confidence score of the LFE channel determination step 120 may be proportional to the difference between the determined sub-band energies in the low frequency region of the LFE channel and the sub-band energies in all other frequency regions, such that relatively larger differences produce more reliable results.

[0145] The LFE channel should include a substantially larger portion of the sub-band sound energy in the low frequency region than in all other frequency regions, so large differences will be more reliable.

[0146] The difference between the sub-band sound energies may be calculated by comparing the sums of the sub-band sound energies in different frequency regions.

[0147] The sum(s) may be further normalized for the size of each frequency bin, respectively.

[0148] Alternatively, the difference between the sub-band sound energies may be calculated by comparing the average values ​​or normalized average values ​​of the sub-band sound energies in different frequency regions.

[0149] The normalized mean will preferably be normalized to the size of each frequency bin.

[0150] The sum is preferred because it results in larger differences, thereby producing a more normalized confidence score.

[0151] The low frequency region may be, for example, any subband below 400 Hz, 300 Hz, 200 Hz, 120 Hz, 100 Hz or 50 Hz.The low frequency region may be determined based on the content of the audio signal.

[0152] In a further embodiment, the confidence score of the LFE channel determination step 120 is proportional to the sum of the sub-band acoustic energies of the determined LFE channel in the frequency region above the frequency threshold, so that a relatively lower sum produces a more reliable result.

[0153] In this embodiment, content in the low frequency region is not used when determining the confidence score. Depending on the embodiment, this may be beneficial.

[0154] In one embodiment, the confidence score of the LFE channel determination step 120 is proportional to: the difference between the determined sub-band acoustic energy in the low frequency region of the LFE channel and the sub-band acoustic energy in all other frequency regions, such that a relatively larger difference produces a more reliable result; and the sum of the determined sub-band acoustic energy of the LFE channel in the frequency region above the frequency threshold, such that a relatively lower sum produces a more reliable result.

[0155] In this embodiment, the two metrics deemed most useful are used in combination, possibly weighted in different ways, to produce a highly reliable confidence score.

[0156] The frequency threshold may be, for example, 2000 Hz, 1000 Hz, 500 Hz, 400 Hz, 300 Hz, 200 Hz, 120 Hz, 100 Hz, or 50 Hz, or may be determined based on the content of the audio signal.

[0157] In some embodiments, the confidence score of the LFE channel determination step 120 is proportional to the highest frequency signal present in the determined LFE channel, such that relatively lower highest frequency signals produce more reliable results.

[0158] The presence or absence of an LFE channel may be determined based on an energy threshold. The energy threshold may be adapted to ignore noise or may be so low that it is substantially non-existent, so that any signal present affects the confidence score.

[0159] In these embodiments, only the maximum cutoff frequency is used when determining the confidence score. Depending on the embodiment, this may be beneficial.

[0160] Since the presence of the LFE channel is unknown, a confidence score below the confidence threshold may result in the result of the LFE channel determination step 120 being marked as unreliable, e.g. in short-term memory or as metadata. If a mismatch is detected, or if an incorrect number (e.g. more than one) of center channels and / or LFE channels are identified (possibly even in a later step), this may result in displaying a warning to the user and / or directly re-performing the LFE channel determination step 120, for example.

[0161] The confidence score of the center channel identification step 140 may be proportional to the independence and / or irrelevance of the identified center channel compared to channels of the Y channels not identified as LFE channels, such that relatively high independence and / or irrelevance produces more reliable results.

[0162] The center channel should be independent and / or uncorrelated compared to the Y channels not identified as the LFE channel, so a high degree of independence and / or uncorrelation will be more reliable.

[0163] If multiple calculation options for the confidence score for a particular step of a method are available, they can be applied in a hierarchy.

[0164] Metadata may be used to store the confidence score.

[0165] In general, a result (for any of the identification steps 110 - 150 ) with a confidence score below a confidence threshold may result in restarting the channel identification method 100 , for example using a larger data length.

[0166] FIG. 7A to FIG. 7B A flow chart showing the steps of the method for channel identification is shown. Which checks are performed in which order and the ordering optimization of the method steps are shown in order to minimize calculations. In this embodiment a 5.1 surround sound file format is assumed, however, other formats are possible with slight modifications.

[0167] The first step is an empty channel identification step 110 . The result of this step allows the method to reduce the number of possible configurations of the multi-channel audio signal to one or two options, which are listed after the result of the empty channel identification step 110 .

[0168] The embodiment shown has six channels, however, any other number is possible while adjusting the result for the number of empty channels.

[0169] If the result of the empty channel identification step 110 is that the number of empty channels is five, the last empty channel will be automatically identified as a center channel and then output.

[0170] If the result of the empty channel identification step 110 is that the number of empty channels is three, the identified empty channel is output and the remaining channels are assumed to be L, R, C. Pairs are found using the channel pair division step 130 and the remaining channel will be automatically identified as the center channel and then output together with the pair.

[0171] If the result of the empty channel identification step 110 is that the number of empty channels is one, it is checked again whether the empty channel is mistaken for the LFE channel using the LFE channel identification step 120. If the LFE channel is detected, the LFE channel is output, otherwise the empty channel is output. Two pairs are found from the five remaining channels using the channel pair division step 130, and the remaining channels will be automatically identified as center channels and then output together with the pairs.

[0172] If the result of the empty channel identification step 110 is that the number of empty channels is zero, then the LFE channel must be present if the input is formatted according to 5.1 surround sound. In an embodiment where, for example, 7.1 formatting is possible, the six remaining channels may be, for example, three pairs. The LFE channel is identified and output using the LFE channel identification step 120. Two pairs are found from the five remaining channels using the channel pair division step 130, and the remaining channel will be automatically identified as the center channel and then output with the pair.

[0173] If the result of the empty channel identification step 110 is that the number of empty channels is two, the identified empty channel is output and the remaining channels can be L, R, C, LFE or L, R, Ls, Rs. Since the LFE channel identification step 120 is relatively efficient, it is used next. If the LFE channel is detected, the LFE channel is output and the remaining channels are L, R, C. Otherwise, the remaining channels are L, R, Ls, Rs. The channel pair division step 130 is used to find one or two pairs from the three or four remaining channels, and any remaining channel will be automatically identified as a center channel. In either case, the identified channels are then output.

[0174] If the result of the empty channel identification step 110 is that the number of empty channels is four, the identified empty channels are output and the remaining channels may be L, R or C, LFE. Since the LFE channel identification step 120 is relatively efficient, it is used next. If the LFE channel is detected, the remaining channel will be automatically identified as the center channel and then output together with the LFE channel. If the LFE channel is not detected, the remaining channels are the L, R pair. The pair may be output directly, or the channel pair division step 130 may be used as a precaution before outputting the divided pairs.

[0175] If the result of the empty channel identification step 110 is that the number of empty channels is six, then all channels are empty. In this case, the empty channels are output and the method ends.

[0176] The illustrated embodiment does not include the channel pair distinguishing step 150. If included, the channel pair distinguishing step 150 would occur before the "output L, R, C, (Ls, Rs)" result.

[0177] The illustrated embodiment does not include a center channel identification step 140 that identifies any single remaining channel as other than the center channel, however it would be straightforward for one skilled in the art to modify this based on the previously discussed embodiments. It is further assumed that any single remaining channel is C rather than LFE, as this is more common, however the LFE channel determination step 120 and / or center channel identification step 140 may be performed in other embodiments that do not assume this.

[0178] Figure 8 The system architecture of the channel order detector 1 is shown. The channel order detector applies the method for channel identification according to the present invention in order to detect the order of channels.

[0179] The channel order detector 1 may be adapted to perform the method according to a computer program product. The computer program product comprises a non-transitory computer readable storage medium having instructions adapted to perform the method according to the invention when executed by a device having processing capabilities, such as a channel order detector.

[0180] A multi-channel audio signal comprising X>1 channels is input 801 into a channel order detector. The segment length 802 of the audio signal may be analyzed from the audio signal or input separately. The segment length 802 corresponds to the total length (in minutes) of the input data. Thus, if an audio file is input, the segment length 802 corresponds to the total length of the audio signal of the file.

[0181] The method for channel identification produces identified channels. The sequence detector may then use the identified channels to output an ordered array 810 of channel labels.

[0182] Any number of confidence scores as previously discussed relating to the reliability of the results of the method may also be output 820. The confidence scores may be normalized to 0 to 1, where a confidence score of 0 represents unreliable and 1 represents reliable, or vice versa.

[0183] The playback system can use the output array of the detected tags to correctly match the multiple channels to the multiple sound sources, so that, for example, the center channel comes out of the center speaker, and so on.

[0184] The system comprising the channel order detector may further comprise a display. The method may comprise a displaying step 160 , wherein the calculated confidence score(s) are displayed on the display 60 .

[0185] Display 60 is beneficial because the user can receive feedback regarding the reliability of the method.

[0186] The displaying step 160 may further include displaying a warning if the calculated confidence score is below a confidence threshold.

[0187] A warning is beneficial because it can alert the user to take action, such as to stop the method, redo the method, or improve the method by, for example, increasing the bitstream rate and / or fixing upstream glitches.

[0188] The identified channel layout may be displayed in a display step 160 (see Figure 5). This can provide more relevant feedback to users.

[0189] In some embodiments, the displaying step 160 further comprises waiting for user input using a user interface such as buttons or a touch screen. The display 60 may therefore comprise interface(s) for receiving such user input.

[0190] This prevents the method from proceeding without the possibility for the user to analyze the results and provide feedback.

[0191] The identified channel layout is approved by the user before being applied to the multi-channel audio signal. This reduces the risk of any errors in application.

[0192] The user may not be prompted to approve an identified channel layout that is identical to the user's setup layout. Since this scenario does not require any changes to the playback system, this saves time and reduces user requirements.

[0193] The display step 160 may further include displaying a warning if the identified channel layout is different from the user's setup layout. Since this may require and / or force a change in the setup layout, the user may wish to know before this happens.

[0194] The warning level may be proportional to the calculated confidence score(s). A confidence score indicating an unreliable result may, for example, warrant: a more noticeable warning so that the user can stop the method, re-execute the method, and / or improve the method; or a less noticeable warning so that the user ignores a possible false warning.

[0195] The displaying step 160 may further include allowing the user to manipulate the displayed data. The user may have information beyond what is available to the method and may add and / or change the data available to the method.

[0196] The manipulated data may be used in the channel identification step of the method. This means that changes made while the method is running may be used to improve the channel identification step that occurs. The manipulated data may additionally or alternatively be used in subsequent runs of the method.

[0197] The display step 160 may further comprise allowing the user to select at least one segment of the signal to be ignored. This allows the user to, for example, identify defects of the interfering method in the audio signal and remove them.

[0198] Fig. 9A diagram showing steps of a method for channel identification. The illustrated embodiment shows different steps of the method being performed in different domains. In this embodiment, the empty channel identification step 110, the LFE determination step 120, the channel pair division step 130 and the center channel identification step 140 occur in the time-frequency domain, such as the wavelet domain; whereas the channel pair distinction step 150 occurs in the spatial domain. This is achieved by, for example, transforming 910, 920 the multi-channel audio signal before certain steps to extract features in certain domains and performing an inverse transform after performing these steps.

[0199] This is just one possible embodiment, in other embodiments method steps different from the ones shown are performed in a domain different from the one shown, or for example the entire method is performed in one domain.

[0200] The method 100 may further include the step of applying 170 the identified channel layout to the multi-channel audio signal. This step may include: changing the channel order of the multi-channel audio signal; redirecting the channels to the identified playback sources, i.e., so that the left channel is output by the left speaker; or any other physical and / or digital manipulation of the multi-channel audio signal to conform to the identified layout as a result of the channel identification method.

[0201] In some embodiments, the identified channel layout is applied only if the calculated confidence score(s) exceed a confidence threshold.

[0202] If the identified channel layout is unreliable, applying the identified channel layout may degrade the projected sound image, so a confidence threshold may be used to prevent this.

[0203] The applying step 170 may include applying the identified channel layout to the multi-channel audio signal using any current metadata.The metadata may make the applying step 170 more efficient and may be used by any further systems in the broadcast chain.

[0204] The channel layout identified by the method may be applied in real time to the multi-channel audio signal when the multi-channel audio signal is streamed to a speaker system.

[0205] Since the proposed method is computationally very efficient, it can be applied in real time without causing any significant delay in the playback.

[0206] The first result may be inaccurate and have a low confidence score, and then as the audio signal plays and more data is acquired, the confidence score increases.

[0207] A real-time embodiment of the method may include: initialization for clearing all data buffers and obtaining the number of channels. After acquiring some new data, channel identification may be performed on all available data. Features of previous data may be used to keep consumption complexity low. Inconsistent data may also be accepted. If a decision cannot be made for specific channels based on the available data, those channels may be marked as unknown and have a confidence score of 0. Initially, all channels have low confidence scores due to a global weighting factor. After enough data is received, the identification remains unchanged and the confidence score may fluctuate slightly.

[0208] The multi-channel audio signal can be a multi-channel surround sound file or stream used in content creation, analysis, transformation and playback systems. These systems are greatly affected by the channel layout.

[0209] At least one step of the method can use a machine learning-based method. The machine learning-based method can be a decision tree, Adaboost, GMM, SVM, HMM, DNN, CNN and / or RNN.

[0210] Machine learning may be used to further improve the efficiency and / or reliability of the method.

[0211] SVM for channel pair detection can be used as an example. The inter-channel spectral distance between channels i and j in frame l is represented as D i，j (l), as shown in Formula 3. Then the entire frequency band is divided into 1, 2, ... or K different frequency bands, and the inter-channel spectral distance is calculated, so as to obtain the average inter-channel spectral distance respectively. Then you can The K values ​​of are grouped into channel distance vectors for channels i and j. For all channels that are not detected as LFE or null, the channel distance vector between each possible pair is calculated. The label of this vector is 1 if channels i and j belong to a pair, otherwise it is 0. A support vector machine can be trained based on a labeled training database and then used to detect channel pairs.

[0212] Fig.10 A flow chart of the channel pair division step 130 is shown. For greater efficiency, channel pair detection is usually performed on non-null and non-LFE channels. If the number of unknown channels is two or more, channel pairs may be detected.

[0213] Matching of symmetrical channels in the channel pair division step 130 may further include comparing temporal features, spectral features, auditory features and / or features in other domains to calculate the acoustic energy distribution and variance between the audio signals of each channel, and matching the most symmetrical channels into pairs.

[0214] Symmetrical channels are found as audio channels with substantially similar and / or symmetrical sound signal content by analyzing the sound energy distribution and variance. Symmetrical sound signal content may, for example, include similar background sounds and different foreground sounds, similar fundamental tones and different treble tones, or vice versa. Symmetrical sound signal content may further include synchronized sounds, such as different parts of a single chord or sounds that start in one channel and end in another channel.

[0215] If the characteristics of two channels are very close but very different from the characteristics of other channels, or if the correlation between the two channels is higher than that of other channels, the two channels can be classified as a channel pair.

[0216] The matching of symmetrical channels in the channel pair division step 130 may further include calculating 1010 an inter-channel spectral distance between channels using the calculated acoustic energy distribution and variance of each channel in the short, medium and / or long duration; the inter-channel spectral distance is a normalized pairwise measure of the distance between two matching acoustic energy subbands in each channel summed over multiple subbands; and matching the channels that are shortest distances from each other into pairs.

[0217] The distance metric used may be Euclidean distance, Manhattan distance and / or Minkowski distance.

[0218] All of the following examples are in the frequency domain, however other domains are possible. In addition to the embodiments with time-frequency features, features obtained from other methods such as signal transformation or signal analysis theory can also be used to perform, for example, detection and / or confidence score estimation. In addition to the above heuristic rule-based methods, machine learning-based methods such as regression, decision tree, adaboost, GMM, HMM or DNN can also be used, for example, for detection and / or confidence score estimation.

[0219] In one embodiment, the distance between channels i and j in frame l is calculated according to the following formula:

[0220]

[0221] Where i, j are in the range [1, C] and i ≠ j, C is the number of channels, B is the number of frequency bands, b = 1.. B is the index of the frequency band, l = 1.. L is the index of the frame, and E b，i (l) and E b，i (l) is the time-frequency energy in frequency band b of channels i and j.

[0222] The average of the calculated inter-channel spectral distances over time can be calculated and used to pair the channels that have the shortest average distance to each other. This average over time is used to measure the long-term similarity between channels.

[0223] In one embodiment, the average inter-channel distance between channels i and j is calculated according to the following formula:

[0224]

[0225] Where i, j are in the range [1, C] and i≠j, l is in the range [1, L], C is the number of channels, and L is the number of frames.

[0226] The lowest and / or highest inter-channel distances may be used instead of or in addition to the average distance. However, the average is preferred because, although pairs of channels are similar on average, they are not necessarily similar on, for example, every frame.

[0227] In an embodiment with inter-channel spectral distances, the center channel identification step 140 may further include analyzing the calculated inter-channel spectral distances of any remaining unpaired channels in the Y channels that are not identified as LFE channels or divided into pairs to identify the center channel. This will further increase the accuracy of the center channel identification step 140.

[0228] The confidence score of the center channel identification step 140 may be proportional to the calculated inter-channel spectral distance between the identified center channel and the other channels of the Y channels that are not identified as LFE channels, so that relatively symmetrical distances produce more reliable results.

[0229] The center channel preferably has a symmetrical distance from other channels (ie, paired channels) that are not identified as LFE channels, so relatively symmetrical distances produce more reliable results.

[0230] The confidence score of the center channel identification step 140 may be proportional to the confidence score of the channel pair segmentation step 130 (if present).

[0231] If, for example, the center channel identification step 140 merely comprises identifying any remaining channels, the reliability of the center channel identification step 140 is directly proportional to the reliability of the channel pair division step 130. In even other embodiments, the reliability of the matching of the pairs may directly affect the reliability of the center channel identification step 140, as this may affect the available channels to be identified as the center channel.

[0232] The matching of symmetrical channels in the channel pair division step 130 may further include comparing the correlation of the acoustic energy distribution of each channel and matching the most correlated channels into pairs. This is a simple and efficient calculation; however, this only works in some embodiments.

[0233] The correlation measure used may be cosine similarity, Pearson correlation coefficient and / or correlation matrix.

[0234] The channel pairing step 130 may further include, for each of the Y channels not identified as an LFE channel, measuring and / or importing from previous measurements (if any) at least one parameter for calculation of matching the channels into pairs.

[0235] The measurement result may be, for example, the acoustic energy measured in the empty channel identification step 110 or the LFE channel determination step 120. This increases the efficiency of the method 100.

[0236] If channel pairs are matched differently depending on the feature(s) used to match them, the hierarchy of the feature(s) may be used to determine which pairings to apply.

[0237] A hierarchy may be, for example, a type of metric that is preferred over another metric, such as average inter-channel spectral distance over maximum inter-channel spectral distance or correlation of acoustic energy distribution.

[0238] The channel pair division step 130 may continue to pair any unpaired channels of the Y channels that are not identified as LFE channels until fewer than two channels remain.

[0239] There may be more than two pairs of channels, such as the front and rear pairs in a 5.1 audio format. Therefore, if there are more than two channels remaining, there may be more channel pairs, and there may be more pairs to divide.

[0240] The channel pair division step 130 may further include assigning the first received channel of the multi-channel audio signal in each pair as a left channel and assigning the last listed channel in each pair as a right channel.

[0241] In multi-channel audio signals, it is customary to list the left channel before the right channel in each pair, so by assuming that this is always the case, method 100 is more efficient.

[0242] Metadata may be used to store the division of channel pairs and / or the assignment of left and right channels (if any).

[0243] The confidence score of the channel pair segmentation step 130 may be proportional to the symmetry metric of the matched pair(s), such that a relatively high symmetry metric produces a more reliable result.

[0244] Correctly matched pairs preferably have high symmetry, so if the result of the channel pair division step 130 has pairs with relatively high symmetry, the result is relatively reliable.

[0245] The confidence score of the channel pair segmentation step 130 may be proportional to the calculated inter-channel spectral distance between the matched pair(s), such that relatively shorter distances produce more reliable results.

[0246] Correctly matched pairs are preferably relatively close to each other, so if the result of the channel pair division step 140 has pairs with relatively short distances, the result is relatively reliable.

[0247] The confidence score of the channel pair segmentation step 140 may be proportional to the calculated inter-channel spectral distance between each channel in the matched pair(s) and other channels of the Y channels that are not identified as LFE channels or matched channels, such that relatively long distances produce more reliable results.

[0248] Correctly matched pairs preferably have longer distances to other channels, so if the result of the channel pair division step 140 has pairs with relatively longer distances to other channels, the result is relatively reliable.

[0249] If the confidence score of the channel pair division step is below the confidence threshold 1030, at least a portion of the channel pair division step may be re-performed 1040 with a different sub-band division when calculating the inter-channel spectral distance.

[0250] By changing the subband division, a more reliable result may be achieved. In some embodiments, the subband division is changed until a satisfactory reliability of the channel pair division step 140 is achieved, for example by a confidence threshold or pairing score threshold 1030 .

[0251] The pairing score is a measure for comparing the likelihood that members of a pair may be grouped into other pairs. The pairing score threshold is a predetermined threshold for the pairing score(s). If the pairing score(s) are above the pairing score threshold, the result of the channel pair segmentation step 140 is sufficiently reliable.

[0252] This version is in Fig.10 1020. The method of claim 104 is shown in the flowchart of . First, the average inter-channel spectral distance is calculated for each possible pair. Then, a pairing score is calculated 1020 for the pair with the lowest inter-channel spectral distance. If the pairing score is not high enough to make a decision, a different time-frequency segmentation can be used to obtain a new average inter-channel spectral distance and a corresponding pairing score. Trials can be performed until all channels are paired or some termination condition is met. If more than two channels are still not detected, their confidence scores are all set to 0.

[0253] The confidence score may be further weighted by a global weight factor to account for the total length of the data. Channel pair detection is performed on all unknown channels until only one channel remains.

[0254] The pairing score may be used as the confidence score or as a part of the confidence score.

[0255] In one embodiment, the pairing score for a pair of channels i and j is calculated according to the following formula:

[0256]

[0257] Among them, M q，i (l) is where D q，i (l)<D i，j (l) where q is the channel index, q≠i, q≠j. q，i The range of (l) is [0, L].

[0258] The pairing score can be calculated for any possible pair or just for the two channels with the lowest average inter-channel spectral distance (ie channels i and j in the above equation).The pairing score is a measure of the confidence of its classification as a channel pair.

[0259] The pairing score compares the inter-channel spectral channel distance between the candidate channel pair i, j and each of the other channels and ensures that the two channels are similar to each other and different from any other channel. If there are other channels that are also similar to channel i or j, then P i，j will be much smaller than 1 and therefore indicate low reliability.

[0260] Fig.11 A flow chart is shown for a channel pair position distinguishing step 150. The channel pair distinguishing step 150 includes distinguishing channels divided into pairs between front pairs, side pairs, rear pairs, and / or any other position pairs.

[0261] The channel pair distinguishing step 150 is a part of the method for channel identification and is preferably performed after the channel pair dividing step 130 .

[0262] Many multi-channel audio signals comprise more than one channel pair; such as 5.1, which comprises a front pair and a rear pair. It would therefore be beneficial for a method for channel identification to be able to distinguish between positional pairs and correctly identify them as such.

[0263] Directional stability of the front sound image is usually maintained for most of the duration, and the rear channels usually carry information that can enhance the overall sound image.

[0264] The channel pair distinguishing step 150 may include calculating 1120 an inter-pair level difference for each pair; the inter-pair level difference being proportional to the decibel difference of the sum of the sub-band acoustic energies for each pair; wherein the pair having the relatively highest level is distinguished as the front pair.

[0265] Alternatively or additionally, amplitude panning may occur in conjunction with the calculation of the inter-pair level differences.Amplitude panning comprises generating a virtual sound source.

[0266] Most virtual sound sources can be generated to appear from the front. This will cause the front pair to have a relatively higher amplitude than other position pairs, so the pair with the highest amplitude can be distinguished as the front pair.

[0267] The translation method may further include out-of-phase the rear pair. Thus, relatively out-of-phase pairs may be distinguished as rear pairs.

[0268] The front pair is traditionally the pair 1140 with the relatively highest level, since the highest level should be closest to the center channel.

[0269] In one embodiment, for each time-frequency slice, the inter-pair level difference between a channel pair i and j and another channel pair m and n of frequency band b is calculated according to the following formula:

[0270]

[0271] Among them, E b，i (l)+E b，j (l) and E b，m (l)+E b，n (l) are the subband energies of the pair (i, j) and the pair (m, n) on frequency band b in frame l, respectively, and E b，i (l) E b，j (l) E b，m (l) and E b，n (l) are the subband energies of frequency band b of channels i, j, m, n in frame l, respectively, where i, j, m, n are unequal integers in the range [1, C], where C is the number of channels; b = 1..B, where B is the number of frequency bands, and l = 1..L, where L is the number of frames.

[0272] The inter-pair level difference between the pairs is not always high enough, as a difference below 2 dB may not be informative. Therefore, signal segments may be selected that have content that may produce a larger inter-pair level difference between the pairs.

[0273] Thus, the channel pair distinguishing step 150 may further comprise selecting one or more segments of the signal for each channel in each pair where the subband acoustic energy of the signal is above the energy threshold; and using only these segments to calculate the inter-pair level difference of the channels.

[0274] By selecting segments having a large amount of information in the form of sub-band acoustic energy exceeding an energy threshold, the inter-pair level difference may be increased.

[0275] The channel pair distinguishing step 150 may further comprise selecting 1150 one or more segments of the signal for each pair where the absolute inter-pair level difference is above an absolute threshold; and using only these segments for calculating the inter-pair level difference of the channels.

[0276] By selecting segments with a high threshold, the average inter-pair level difference can be increased. Many multi-channel audio signals have similar outputs in more than one channel during parts of the signal. These parts do not contribute to the inter-pair level difference and can therefore be safely ignored.

[0277] In addition to or instead of measuring absolute inter-pair level differences, the average inter-pair level differences in segments that are relatively small compared to the total length of the signal may also be used.

[0278] If selection of segments does not result in a sufficiently high mean inter-pair level difference, selection with a higher absolute threshold may achieve this.

[0279] Thus, if the relative highest average inter-pair level difference is below the level threshold (determined in step 1130), the step of calculating the inter-pair level differences for the channels may be repeated with a higher absolute threshold 1150 until the average inter-pair level differences are sufficiently high.

[0280] Alternatively or additionally, the pair having the relatively highest combined directional coherence with the identified center channel may be distinguished as the leading pair if the relatively highest average inter-pair level difference is below a level threshold.

[0281] In one embodiment, selection of segments is abandoned and directional consistency with the identified center channel may be used instead to distinguish the pairs. The pair with the direction closest to the center channel is also closest to the center channel.

[0282] Directional coherence is a measure of the similarity of two channels in the time domain, which is related to the image direction, which in turn means the phase difference between the channels.

[0283] The directional difference can be used to measure the directional consistency of the main sound source between the two channels. A simplified measurement of directional consistency according to an embodiment is as follows:

[0284]

[0285] Among them, S i (n) is the nth sample value of channel i in the time domain, such that S i Each value of (n) corresponds to a point on the waveform, and the total sample value is T. It means the phase difference between the two channels.

[0286] The front pairs should traditionally have relatively higher directional consistency with each other than the other position pairs, and the rear pairs should traditionally have relatively lower directional consistency with each other than the other position pairs.

[0287] The signals in the front pair are usually time aligned to represent directional sound sources, so they have higher correlation and lower delay. This means that there are more identical components in the front pair compared to the rear pair. The directional difference is used to measure this, as illustrated in equation 7. If the signals in channels i and j are the same, this means that the two channels are in phase and then X=1, otherwise, X<1. If the two channels are out of phase, then X=0.

[0288] In another embodiment, the pair with the relatively highest combined directional coherence with the identified center channel 1170 is distinguished as the front pair 1180 if the relative highest average inter-pair level difference is below the level threshold and the absolute threshold is above the maximum threshold 1160 .

[0289] This embodiment Fig.11 . In this embodiment, all signals are first selected 1110, however the average inter-pair level difference does not reach a high enough level to exceed the level threshold, and the selection of segments fails to produce a high enough average inter-pair level difference. Therefore, the directional consistency with the identified center channel can be used instead to distinguish these pairs.

[0290] The selection of segments failed because the average inter-pair level difference did not reach a high enough level to exceed the level threshold, and the absolute threshold was so high that the segments exceeding this absolute threshold were not long enough to calculate the inter-pair level difference.

[0291] The level threshold may be a constant between 2-3 dB. The maximum threshold of the absolute threshold may be 2 dB and / or any threshold that results in the total length of the selected segments being shorter than, for example, 20% of the length of the non-silent signal or shorter than, for example, 1 minute.

[0292] The maximum threshold of the absolute threshold relates to the case when the selected segment or segments of the signal of each channel in each pair in which the average inter-channel spectral distance is above the distance threshold are no longer long enough to calculate the inter-channel level difference. If the total length of the selected segments is shorter than 20% of the length of the non-silent signal or shorter than, for example, 1 minute, the useful signal is too short.

[0293] The distinction between the position pairs can be based on the similarity of these position pairs to the identified center channel. In this case, the pair that is most similar to the identified center channel can be distinguished as the front pair, and the pair that is least similar to the identified center channel can be distinguished as the back pair.

[0294] Traditionally the center channel is the front of the sound image, so the front pair should be more like the center channel than the rear pair, for example.

[0295] The similarity to the identified center channel may be based on time-frequency characteristics, spatial characteristics, image direction, phase differences between channels, and / or inter-channel pair level differences.

[0296] Alternatively or additionally, delay panning may be used to calculate similarity to the identified center channel, wherein the pair with the highest directional consistency with the center channel is distinguished as the front pair.

[0297] Time-frequency features are examined first, followed by spatial features, because amplitude shift is the most commonly used and the calculation of time-frequency features is not very time-consuming.

[0298] A directional pattern of channels may be generated to compare the center-to-pair distances of channel pairs. Then, the channel pair that is closer to the center channel is detected as the front pair.

[0299] If different pairs are distinguished as identical position pairs based on features used for distinguishing, the features may be prioritized based on the hierarchy.

[0300] The tiers may depend on, for example, the confidence score, the metric used, or the threshold used.

[0301] Metadata may be used to store the distinction between channel pairs.

[0302] A confidence score may be calculated for the results of the channel pair distinguishing step 150 .

[0303] The confidence score of the channel pair distinguishing step 150 may be proportional to the calculated inter-channel spectral distance between the identified center channel and pairs of channels in the Y channels that are not identified as LFE channels, such that a relatively small inter-channel spectral distance between the front pair and the center channel produces a more reliable result.

[0304] The pairs that are closest to the identified center channel should be distinguished as front pairs, and the pairs that are least similar to the identified center channel should be distinguished as back pairs, and the metric reflects this.

[0305] The confidence score of the channel pair distinguishing step 150 may be proportional to the directionality of the divided pairs of channels, so that relatively large differences between the directionality produce more reliable results.

[0306] The pair with a direction closer to the center channel is also closer to the center channel and is therefore the front pair. Therefore, a large difference produces a more reliable distinction. The absolute differences and / or ratios of different pairs can be used.

[0307] For similar reasons, the confidence score of the channel pair distinguishing step 150 may be proportional to the directionality of the identified center channel and the channels of the divided pair, so that a relatively small difference between the directionality of the center channel and one of the pair produces a more reliable result.

[0308] The confidence score of the channel pair distinguishing step 150 may be proportional to the calculated inter-pair level differences of the paired channels, so that a relatively high average level difference produces a more reliable result.

[0309] Average inter-pair level differences above 2 dB provide useful information, and the higher the average inter-pair level difference, the more useful information is provided. More information produces more reliable results.

[0310] The confidence score of the channel pair distinguishing step 150 may be proportional to the confidence score of the channel pair dividing step 130 and / or the center channel identifying step 140 (if present).

[0311] If the channel pair division step 130 is unreliable, then the channel pair differentiation step 150 will also be unreliable. Further, many possible confidence score calculations for the channel pair differentiation step 150 depend on the center channel identification step 140. Therefore, to save computation, confidence scores previously calculated for the channel pair division step 130 and / or the center channel identification step 140 may be reused.

[0312] The confidence score of the channel pair distinguishing step 150 may be proportional to the length of the selected segment or segments of the signal, such that relatively long segment or segments produce more reliable results.

[0313] A short length of the selected segment will make the calculation of the inter-pair level difference unreliable.The absolute length of the selected segment and / or the ratio of the length of the selected segment to the total length of the data may be used.

[0314] If the confidence score of the channel pair distinguishing step 150 is below the confidence threshold, at least a portion of the channel pair distinguishing step 150 may be re-performed with a different data segment.

[0315] This ensures that the result of the channel pair distinguishing step 150 is reliable.

[0316] After studying the above description, further embodiments of the present disclosure will become apparent to those skilled in the art. Although the present specification and the accompanying drawings disclose embodiments and examples, the present disclosure is not limited to these specific examples. Many modifications and variations may be made without departing from the scope of the present disclosure as defined by the appended claims. Any reference numerals appearing in the claims should not be construed as limiting their scope.

[0317] Additionally, variations of the disclosed embodiments may be understood and effected by those skilled in the art in practicing the present disclosure by studying the drawings, the present disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0318] The systems and methods disclosed above may be implemented as software, firmware, hardware, or a combination thereof. For example, aspects of the present application may be at least partially embodied in an apparatus, a system including more than one device, a method, a computer program product, etc. In a hardware implementation, the division of tasks between the functional units mentioned in the above description does not necessarily correspond to the division of physical units; on the contrary, a physical component may have multiple functions, and a task may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a digital signal processor or a microprocessor or as hardware or an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transient medium) and a communication medium (or transient medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. Further, it is well known to those skilled in the art that communication media typically implement computer-readable instructions, data structures, program modules or other data in the form of a modulated data signal such as a carrier wave or other transmission mechanism, and includes any information delivery media.

[0319] Various aspects of the present invention may be understood from the following enumerated example embodiments (EEE):

[0320] EEE 1. A method for performing channel identification on a multi-channel audio signal including X>1 channels, the method (100) comprising the following steps:

[0321] identifying (110) any empty channels among the X channels, thereby producing a subset having Y≤X non-empty channels;

[0322] determining (120) whether a low frequency effects (LFE) channel exists in the Y channels, and when it is determined that the LFE channel exists, identifying the determined channel in the Y channels as the LFE channel;

[0323] dividing (130) the remaining channels of the Y channels not identified as the LFE channels into any number of channel pairs by matching symmetrical channels; and

[0324] Any remaining unpaired channels of the Y channels that are not identified as the LFE channel or divided into pairs are identified (140) as center channels.

[0325] EEE 2. The method according to EEE 1, further comprising the step of distinguishing (150) the channels divided into pairs between front pairs, side pairs, rear pairs and / or any other position pairs.

[0326] EEE 3. The method according to EEE 2, wherein the channel pair distinguishing step includes calculating the inter-pair level difference between the pairs; the inter-pair level difference is proportional to the decibel difference of the sum of the sub-band sound energy of each pair; wherein the pair with the relatively highest level is distinguished as the front pair.

[0327] EEE 4. The method according to EEE 3, wherein the channel pair distinguishing step further comprises amplitude panning combined with the calculation of the inter-pair level difference, the amplitude panning comprising generating a virtual sound source. ...

[0328] EEE 5. The method according to EEE 3 or 4, wherein the channel pair distinguishing step further comprises selecting one or more segments of the signal for each pair in which the sub-band acoustic energy of the signal is above an energy threshold; and using only these segments to calculate the inter-pair level difference of the pair.

[0329] EEE 6. The method according to any one of EEE 3 to 5, wherein the channel pair distinguishing step further comprises selecting one or more segments of the signal in each pair in which the absolute inter-pair level difference is above an absolute threshold; and using only these segments to calculate the inter-pair level difference. ...

[0330] EEE 7. The method according to EEE 6, wherein if the relatively highest average inter-pair level difference is below a level threshold, the step of calculating the inter-pair level differences of the channels is repeated with a higher absolute threshold. ...

[0331] EEE 8. A method according to any one of EEEs 3 to 7, wherein the pair having the relatively highest combined directional consistency with the identified center channel is distinguished as the front pair if the relatively highest average inter-pair level difference is below a level threshold. EEE 8. A method according to any one of EEEs 3 to 7, wherein the pair having the relatively highest combined directional consistency with the identified center channel is distinguished as the front pair.

[0332] EEE 9. A method according to EEE 7, wherein the pair having the relatively highest combined directional consistency with the identified center channel is distinguished as the front pair if the relatively highest average pair-to-pair level difference is below a level threshold and the absolute threshold is above a maximum threshold.

[0333] EEE 10. The method according to EEE 9, wherein a maximum threshold of the absolute threshold is 2 dB.

[0334] EEE 11. A method according to any one of EEE 8 to 10, wherein the directional consistency is a measure of the similarity of two channels in the time domain, the directional consistency is related to the sound image direction, and the sound image direction in turn means the phase difference between the channels.

[0335] EEE 12. The method according to any one of EEE 7 to 11, wherein the level threshold is a constant between 2-3 dB. EEE 13. The method according to any one of EEE 7 to 11, wherein the level threshold is a constant between 2-3 dB.

[0336] EEE 13. A method according to any one of EEE 2 to 12, wherein the distinction between the position pairs is based on their similarity to the identified center channel.

[0337] EEE 14. The method according to EEE 13, wherein the pair most similar to the identified center channel is distinguished as the front pair, and the pair least similar to the identified center channel is distinguished as the back pair. EEE 15. The method according to EEE 15, wherein the pair most similar to the identified center channel is distinguished as the back pair.

[0338] EEE 15. The method according to EEE 13 or 14, wherein the similarity to the identified center channel is based on time-frequency characteristics, spatial characteristics, sound image direction, phase difference between the channels and / or inter-pair level difference.

[0339] EEE 16. A method according to any one of EEE 13 to 15, wherein the similarity to the identified center channel is calculated using delay panning, wherein the pair with the highest directional consistency with the center channel is distinguished as the front pair.

[0340] EEE 17. A method according to any one of EEE 13 to 16, wherein the similarity to the identified center channel is calculated by generating a directional pattern of the channels to compare the center-to-pair distances of the channel pairs, wherein the pair closer to the center channel is distinguished as the front pair.

[0341] EEE 18. The method according to any one of EEE 2 to 17, wherein if different pairs are distinguished as the same position pair depending on the features used for distinguishing, the features are prioritized according to a hierarchy. EEE 18. The method according to any one of EEE 2 to 17, wherein the features are prioritized according to a hierarchy.

[0342] EEE 19. The method according to any one of EEE 2 to 18, wherein metadata is used to store the distinction of the channel pairs. ...

[0343] EEE 20. The method according to any one of the preceding EEEs, wherein the empty channel identification step further comprises measuring the acoustic energy in each of the X channels.

[0344] EEE 21. The method according to EEE 20, wherein the acoustic energy in each of the X channels is measured in short, medium and / or long duration. EEE 22. The method according to EEE 22, wherein the acoustic energy in each of the X channels is measured in short, medium and / or long duration.

[0345] EEE 22. The method according to EEE 20 or 21, wherein a channel is identified as empty if the total acoustic energy of the channel is below an energy threshold. EEE 23. The method according to EEE 23 or 21, wherein the channel is identified as empty if the total acoustic energy of the channel is below an energy threshold.

[0346] EEE 23. The method according to any one of EEEs 20 to 22, wherein a channel is identified as empty if the acoustic energy of each subband of the channel is below an energy threshold. EEE 24. The method according to any one of EEEs 20 to 22, wherein the channel is identified as empty if the acoustic energy of each subband of the channel is below an energy threshold.

[0347] EEE 24. The method according to any one of EEE 20 to 23, wherein the acoustic energy is measured in the time domain, spectral domain, wavelet domain and / or auditory domain.

[0348] EEE 25. The method according to any one of the preceding EEEs, wherein metadata is used to store identification of empty channels.

[0349] EEE 26. The method according to any one of EEEs 20 to 25, wherein the LFE channel determining step further comprises using acoustic energy measured in each of the Y channels to determine whether the LFE channel exists. ...

[0350] EEE 27. The method according to any one of the preceding EEEs, wherein the LFE channel determination step further comprises measuring a frequency band in each of the Y channels where acoustic energy above an energy threshold exists.

[0351] EEE 28. The method according to EEE 27, wherein the frequency band in each of the Y channels where there is acoustic energy above an energy threshold is measured in short, medium and / or long duration. EEE 28. The method according to EEE 27, wherein the frequency band in each of the Y channels where there is acoustic energy above an energy threshold is measured in short, medium and / or long duration.

[0352] EEE 29. A method according to any one of EEEs 26 to 28, wherein if the sum of sub-band sound energies in a low frequency region of a channel is significantly higher than the sum of sub-band sound energies in all other frequency regions in the channel, it is determined that an LFE channel exists in the Y channels.

[0353] EEE 30. The method according to EEE 29, wherein the sum of the sub-band acoustic energies in each frequency region is further normalized according to the size of each frequency region. EEE 31. The method according to EEE 32.

[0354] EEE 31. A method according to EEE 29 or 30, wherein any such channel is identified as the LFE channel.

[0355] EEE 32. The method according to any one of EEE 29 to 31, wherein the low frequency region includes any sub-band below 200 Hz. ...

[0356] EEE 33. A method according to any one of EEE 26 to 32, wherein if the channel includes only sub-band sound energy above an energy threshold in a frequency region below a frequency threshold, it is determined that an LFE channel exists in the Y channels.

[0357] EEE 34. The method according to EEE 33, wherein only any such channel is identified as the LFE channel. EEE 34. The method according to EEE 33, wherein only any such channel is identified as the LFE channel.

[0358] EEE 35. The method according to EEE 33 or 34, wherein the frequency threshold is 200 Hz or higher.

[0359] EEE 36. A method according to any one of EEE 26 to 35, wherein, if it is determined that several LFE channels exist among the Y channels, only one LFE channel is identified as the LFE channel according to a hierarchy of (multiple) features used to determine whether an LFE channel exists.

[0360] EEE 37. A method according to any one of the preceding EEEs, wherein metadata is used to store the identification of the LFE channel.

[0361] EEE 38. A method according to any one of the preceding EEEs, wherein the matching of symmetrical channels in the channel pair division step further includes comparing temporal features, spectral features, auditory features and / or features in other domains to calculate the acoustic energy distribution and variance between the audio signals of each channel, and matching the most symmetrical channels into pairs.

[0362] EEE 39. A method according to EEE 38, wherein the matching of symmetrical channels in the channel pair division step further includes calculating the inter-channel spectral distance between the channels using the acoustic energy distribution and variance calculated for each channel in the short, medium and / or long durations; the inter-channel spectral distance is a normalized pairwise measure of the distance between two matching acoustic energy subbands in each channel summed over multiple subbands; and matching the channels with the shortest distances to each other into pairs.

[0363] EEE 40. The method according to EEE 39, wherein the distance metric used is Euclidean distance, Manhattan distance and / or Minkowski distance.

[0364] EEE 41. The method according to EEE 38 or 40, wherein an average value of the calculated inter-channel spectral distances over time can be calculated and used to pair channels having the shortest average distance from each other. EEE 42. The method according to EEE 42 or 42, wherein an average value of the calculated inter-channel spectral distances over time can be calculated and used to pair channels having the shortest average distance from each other.

[0365] EEE 42. A method according to any one of EEE 39 to 41, wherein the center channel identification step further comprises analyzing the calculated inter-channel spectral distance of any remaining unpaired channels of the Y channels that are not identified as the LFE channel or divided into pairs to identify the center channel.

[0366] EEE 43. The method according to any one of the preceding EEEs, wherein the matching of symmetrical channels in the channel pair division step further comprises comparing the correlation of the sound energy distribution of each channel and matching the most correlated channels into pairs. EEE 44.

[0367] EEE 44. The method according to EEE 43, wherein the correlation measure used is cosine similarity, Pearson correlation coefficient and / or correlation matrix.

[0368] EEE 45. A method according to any one of EEEs 38 to 44, wherein the channel pair division step further includes, for each channel of the Y channels that is not identified as the LFE channel, measuring and / or importing from previous measurement results (if any) at least one parameter of the calculation for matching the channels into pairs.

[0369] EEE 46. A method according to any one of EEE 38 to 45, wherein if the channel pairs are matched differently according to (multiple) features used to match the channel pairs, the hierarchy of (multiple) features used determines which pairings to apply.

[0370] EEE 47. The method according to any one of the preceding EEEs, wherein the channel pairing step continues to pair any unpaired channels in the Y channels that are not identified as the LFE channel until fewer than two channels remain. EEE 47.

[0371] EEE 48. The method according to any of the preceding EEEs, wherein the channel pair division step further comprises assigning a first received channel of the multi-channel audio signal in each pair as a left channel and assigning a last listed channel in each pair as a right channel. EEE 48. The method according to any of the preceding EEEs, wherein the channel pair division step further comprises assigning a first received channel of the multi-channel audio signal in each pair as a left channel.

[0372] EEE 49. A method according to any of the preceding EEEs, wherein metadata is used to store the division of channel pairs and / or the allocation of left and right channels (if any).

[0373] EEE 50. A method according to any one of the preceding EEEs, wherein the center channel identification step further comprises calculating the independence and / or irrelevance of any remaining unpaired channels of the Y channels that are not identified as the LFE channels or divided into pairs compared to other channels of the Y channels, and identifying the center channel as the most independent and / or irrelevant channel.

[0374] EEE 51. The method according to EEE 50, wherein the calculation of independence and / or irrelevance of any remaining unpaired channels of the Y channels that are not identified as the LFE channels or divided into pairs is calculated only compared to the channels divided into pairs. EEE 52. The method according to EEE 52, wherein the calculation of independence and / or irrelevance of any remaining unpaired channels of the Y channels that are not identified as the LFE channels or divided into pairs is calculated only compared to the channels divided into pairs.

[0375] EEE 52. A method according to EEE 50 or 51 dependent on at least one of EEEs 2 to 19, wherein the center channel identification step occurs after the channel pair differentiation step, and the independence and / or irrelevance of any remaining unpaired channels of the Y channels that are not identified as the LFE channels or divided into pairs are calculated only compared to the channels differentiated as the front pairs.

[0376] EEE 53. A method according to any one of the preceding EEEs, wherein metadata is used to store the identification of the center channel.

[0377] EEE 54. The method according to any of the preceding EEEs, further comprising calculating a confidence score for any result of the steps of the method, the confidence score being a measure of the reliability of the result.

[0378] EEE 55. The method according to EEE 54, wherein, if the duration of the multi-channel audio signal is below a certain duration threshold, the confidence score is multiplied by a weighting factor less than one, so that a duration less than the duration threshold produces a less reliable result. EEE 55. The method according to EEE 54,

[0379] EEE 56. A method according to EEE 55, wherein the weight factor is proportional to the duration divided by the duration threshold, so that relatively longer durations produce more reliable results.

[0380] EEE 57. The method according to EEE 55 or 56, wherein if the duration is longer than the duration threshold, the weight factor is not applied or the weight factor is equal to one. ...

[0381] EEE 58. A method according to any one of EEE 55 to 57, wherein the duration threshold is a constant between 5-30 minutes.

[0382] EEE 59. A method according to any one of EEE 54 to 58, wherein the confidence score of the empty channel identification step is proportional to the acoustic energy of the identified empty channel, so that relatively lower acoustic energy produces more reliable results.

[0383] EEE 60. A method according to any one of EEE 54 to 59, wherein the confidence score of the LFE channel determination step is proportional to the difference between the sub-band sound energy in the low frequency region of the determined LFE channel and the sub-band sound energy in all other frequency regions, so that a relatively larger difference produces a more reliable result.

[0384] EEE 61. The method according to EEE 60, wherein the difference between the sub-band acoustic energies is calculated by comparing the sums of the sub-band acoustic energies in different frequency regions. EEE 62. The method according to EEE 62, wherein the difference between the sub-band acoustic energies is calculated by comparing the sums of the sub-band acoustic energies in different frequency regions.

[0385] EEE 62. The method according to EEE 60 or 61, wherein the low frequency region includes any sub-band below 200 Hz.

[0386] EEE 63. A method according to any one of EEE 54 to 62, wherein the confidence score of the LFE channel determination step is proportional to the sum of the sub-band acoustic energies of the determined LFE channel in the frequency region above a frequency threshold, so that a relatively lower sum produces a more reliable result.

[0387] EEE 64. The method according to EEE 63, wherein the frequency threshold is 200 Hz or higher.

[0388] EEE 65. A method according to any one of EEE 54 to 64, wherein the confidence score of the LFE channel determination step is proportional to the highest frequency signal present in the determined LFE channel, so that relatively lower highest frequency signals produce more reliable results.

[0389] EEE 66. A method according to any one of EEE 54 to 65, wherein the confidence score of the channel pair division step is proportional to the symmetry measure of the (multiple) matched pairs, so that a relatively high symmetry measure produces a more reliable result.

[0390] EEE 67. A method according to any one of EEE 54 to 66, wherein the confidence score of the channel pair division step is proportional to the calculated inter-channel spectral distance between the (multiple) matched pairs, so that relatively shorter distances produce more reliable results.

[0391] EEE 68. A method according to any one of EEE 54 to 67, wherein the confidence score of the channel pair division step is proportional to the calculated inter-channel spectral distance between each channel in the (multiple) matched pairs and the other channels of the Y channels that are not identified as the LFE channels or matched channels, so that relatively long distances produce more reliable results.

[0392] EEE 69. A method according to any one of EEEs 66 to 68, wherein if the confidence score of the channel pair division step is lower than a confidence threshold, at least a part of the channel pair division step is re-executed with a different sub-band division when calculating the inter-channel spectral distance. EEE 69.

[0393] EEE 70. A method according to any one of EEE 54 to 69, wherein the confidence score of the center channel identification step is proportional to the independence and / or irrelevance of the identified center channel compared to the channels in the Y channels that are not identified as the LFE channels, so that relatively high independence and / or irrelevance produces more reliable results.

[0394] EEE 71. A method according to any one of EEE 54 to 70, wherein the confidence score of the center channel identification step is proportional to the calculated inter-channel spectral distance between the identified center channel and the other channels of the Y channels that are not identified as the LFE channels, so that relatively symmetrical distances produce more reliable results.

[0395] EEE 72. A method according to any one of EEEs 54 to 71, wherein the confidence score of the center channel identification step is proportional to the confidence score of the channel pair division step (if any).

[0396] EEE 73. A method according to any one of EEE 54 to 72 depending on at least one of EEE 2 to 19, wherein a confidence score is calculated for the result of the channel pair distinguishing step.

[0397] EEE 74. A method according to EEE 73, wherein the confidence score of the channel pair distinguishing step is proportional to the calculated inter-channel spectral distance between the identified center channel and the paired channels of the Y channels that are not identified as the LFE channel, so that a relatively small inter-channel spectral distance between the front pair and the center channel produces a more reliable result.

[0398] EEE 75. A method according to EEE 73 or 74, wherein the confidence score of the channel pair distinguishing step is proportional to the directionality of the channels of the divided pairs, so that a relatively large difference between the directionality produces a more reliable result.

[0399] EEE 76. A method according to any one of EEE 73 to 75, wherein the confidence score of the channel pair distinguishing step is proportional to the directionality of the identified center channel and the channels of the divided pairs, so that a relatively small difference between the directionality of the center channel and one of the pairs produces a more reliable result.

[0400] EEE 77. A method according to any one of EEE 73 to 76, wherein the confidence score of the channel pair distinguishing step is proportional to the calculated inter-pair level difference of the channel pairs, so that a relatively high average level difference produces a more reliable result. EEE 77.

[0401] EEE 78. A method according to any one of EEE 73 to 77, wherein the confidence score of the channel pair distinguishing step is proportional to the confidence score of the channel pair dividing step and / or the central channel identifying step (if any).

[0402] EEE 79. A method according to any one of EEE 73 to 78 that is at least subordinate to EEE 4 or 5, wherein the confidence score of the channel pair distinguishing step is proportional to the length of the selected one or more segments of the signal, so that relatively long one or more segments produce more reliable results.

[0403] EEE 80. A method according to any one of EEE 73 to 79, wherein if the confidence score of the channel pair distinguishing step is lower than a confidence threshold, at least a part of the channel pair distinguishing step is re-performed with a different data segment. EEE 81. A method according to any one of EEE 73 to 79, wherein if the confidence score of the channel pair distinguishing step is lower than a confidence threshold, at least a part of the channel pair distinguishing step is re-performed with a different data segment.

[0404] EEE 81. A method according to any one of EEE 54 to 80, wherein, if multiple calculation options for the confidence score of a particular step of the method are available, the calculation options are applied in a hierarchy. EEE 82. A method according to any one of EEE 54 to 80, wherein, if multiple calculation options for the confidence score of a particular step of the method are available, the calculation options are applied in a hierarchy.

[0405] EEE 82. A method according to any one of EEE 54 to 81, wherein the confidence score is stored using metadata.

[0406] EEE 83. The method according to any one of EEE 54 to 82, further comprising a display step (160), wherein the calculated confidence score(s) are displayed on a display (60).

[0407] EEE 84. The method according to EEE 83, wherein the displaying step further comprises displaying a warning if the calculated confidence score is below a confidence threshold. EEE 84.

[0408] EEE 85. The method according to any one of the preceding EEEs, further comprising a display step, wherein the identified channel layout is displayed.

[0409] EEE 86. A method according to any one of EEE 83 to 85, wherein the displaying step further comprises waiting for user input using a user interface such as a button or a touch screen.

[0410] EEE 87. The method according to EEE 85 and 86, wherein the identified channel layout is approved by the user before being applied to the multi-channel audio signal. ...

[0411] EEE 88. A method according to EEE 87, wherein the user is not prompted to approve the identified channel layout that is the same as the user's setup layout.

[0412] EEE 89. A method according to any one of EEE 83 to 88, wherein the displaying step further comprises displaying a warning if the identified channel layout is different from a set layout of the user. EEE 89.

[0413] EEE 90. A method according to EEE 89 as appended to any one of EEE 54 to 82, wherein the warning level is proportional to the calculated confidence score(s).

[0414] EEE 91. A method according to any one of EEE 83 to 90, wherein the displaying step further comprises allowing a user to manipulate the displayed data. EEE 92. A method according to any one of EEE 83 to 90, wherein the displaying step further comprises allowing a user to manipulate the displayed data.

[0415] EEE 92. The method according to EEE 91, wherein manipulated data is used in the channel identification step of the method.

[0416] EEE 93. The method according to any one of EEE 83 to 92, wherein the displaying step further comprises allowing a user to select at least one segment of the signal to be ignored. EEE 93.

[0417] EEE 94. The method according to any one of the preceding EEEs, further comprising the step of applying (170) the identified channel layout to the multi-channel audio signal. EEE 94.

[0418] EEE 95. A method according to EEE 94 as appended to any one of EEE 54 to 82, wherein the identified channel layout is applied only when (multiple) of the calculated confidence scores exceed a confidence threshold.

[0419] EEE 96. The method according to EEE 94 or 95, wherein the applying step comprises applying the identified channel layout to the multi-channel audio signal using any current metadata. ...

[0420] EEE 97. A method according to any one of the preceding EEEs, wherein the channel layout identified by the method is applied to the multi-channel audio signal in real time when the multi-channel audio signal is streamed to a speaker system. EEE 98. A method according to any one of the preceding EEEs, wherein the channel layout identified by the method is applied to the multi-channel audio signal in real time when the multi-channel audio signal is streamed to a speaker system.

[0421] EEE 98. The method according to any one of the preceding EEEs, wherein the multi-channel audio signal is a multi-channel surround sound file or stream for use in a content creation, analysis, transformation and playback system.

[0422] EEE 99. A method according to any one of the preceding EEEs, wherein at least one of the steps of the method uses a machine learning-based method.

[0423] EEE 100. The method according to EEE 99, wherein the machine learning based method is a decision tree, Adaboost, GMM, SVM, HMM, DNN, CNN and / or RNN.

[0424] EEE 101. A device configured for identifying channels of a multi-channel audio signal, the device (1) comprising a circuit configured to perform the method (100) according to any one of the preceding claims.

[0425] EEE 102. A computer program product comprising a non-transitory computer-readable storage medium having instructions adapted to perform a method as described in any one of EEE 1 to EEE 100 when executed by a device (1) having processing capabilities.

Claims

1. A method for performing channel identification on a multi-channel audio signal including X>1 channels, the method comprising the following steps: Identifying any empty channels among the X channels, thereby generating a subset having Y ≤ X non-empty channels; determining whether a low frequency effect LFE channel exists in the Y channels, and when it is determined that the LFE channel exists, identifying the determined channel in the Y channels as the LFE channel; Dividing the remaining channels of the Y channels that are not identified as the LFE channels into any number of channel pairs by matching symmetrical channels; and identifying any remaining unpaired channel of the Y channels that is not identified as the LFE channel or divided into pairs as a center channel, The method further comprises the step of distinguishing the channels divided into pairs between front pairs, side pairs, rear pairs and / or any other position pairs, wherein the channel pair distinguishing step comprises calculating the average inter-pair level difference between every two pairs; the inter-pair level difference for a given time-frequency slice is proportional to the decibel difference between the sum of the sub-band acoustic energies of the channels in the pair; wherein the pair with the relatively highest level is distinguished as the front pair, Wherein, the channel pair distinguishing step further comprises: selecting one or more time segments of the signal for each pair in which the absolute inter-pair level difference is above a first threshold; and using only these segments to calculate the average inter-pair level difference, wherein if the relatively highest average inter-pair level difference is below a second threshold, repeating the step of calculating the inter-pair level difference with a higher first threshold.

2. The method according to claim 1, wherein: If the relatively highest average pair-level difference is lower than the second threshold and the first threshold is higher than the maximum threshold, the pair with the relatively highest directional consistency is distinguished as the front pair, wherein the directional consistency is a measure of the similarity of two channels in the time domain, and the directional consistency is related to the sound image direction, which in turn means the phase difference between the channels.

3. The method according to any one of claims 1 to 2, wherein: The empty channel identification step further includes measuring the acoustic energy in each of the X channels, wherein the channel is identified as empty if the total acoustic energy of the channel is below an energy threshold.

4. The method according to any one of claims 1 to 2, wherein: If the sum of subband sound energies in the low frequency region of a channel, i.e., any subband below 200 Hz, is significantly higher than the sum of subband sound energies in all other frequency regions in the channel, it is determined that an LFE channel exists in the Y channels.

5. The method according to any one of claims 1 to 2, wherein: The matching of the symmetrical channels in the channel pair division step further includes: calculating an inter-channel spectral distance between the channels using the calculated acoustic energy distribution and variance of each channel; the inter-channel spectral distance is a normalized pairwise measure of the distance between two matching acoustic energy subbands in each channel summed over multiple subbands; and matching the channels with the shortest distances to each other into pairs.

6. The method according to any one of claims 1 to 2, wherein: The channel pairing step continues pairing any unpaired channels of the Y channels that are not identified as the LFE channel until fewer than two channels remain.

7. The method according to any one of claims 1 to 2, further comprising: Calculating a confidence score for any result of the steps of the method, the confidence score being a measure of how reliable the result is, wherein if the duration of the multi-channel audio signal is below a certain duration threshold, the confidence score is multiplied by a weighting factor less than one, such that durations less than the duration threshold produce less reliable results.

8. The method according to claim 7, further comprising a display step, wherein: The calculated confidence score is displayed on a display; and wherein a warning is displayed if the calculated confidence score is below a confidence threshold and / or if the identified channel layout is different from a user's set layout.

9. The method according to any one of claims 1 to 2, further comprising the step of applying the identified channel layout to the multi-channel audio signal.

10. The method according to any one of claims 1 to 2, wherein: The channel layout identified by the method is applied to the multi-channel audio signal in real time when the multi-channel audio signal is streamed to a speaker system.

11. The method according to any one of claims 1 to 2, wherein: At least one of the steps of the method uses a machine learning based method, wherein the machine learning based method is a decision tree, Adaboost, GMM, SVM, HMM, DNN, CNN and / or RNN.

12. A device configured for identifying channels of a multi-channel audio signal, the device comprising a circuit configured to perform the method according to any one of claims 1 to 11.

13. A non-transitory computer-readable storage medium having instructions stored thereon, wherein: The instructions are adapted to, when executed by a device having processing capabilities, perform the method of any one of claims 1 to 11.

14. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 11 when executed by a device having processing capabilities.

Images