Content-based spatial remixing
By separating the stereo signal into multiple audio content categories and spatially positioning and remixing, the problems of distortion and artifacts during stereo separation are solved, and a more realistic spatial acoustic experience is achieved.
Patent Information
- Application Number
- CN202210411021.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-04-19
- Filing Date
- 2022-04-19
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-04-19
AI Technical Summary
When the prior art separates stereo audio content into multiple audio content categories, distortion and artifacts are easily generated, and the separated audio content cannot be effectively positioned and remixed in space, affecting the spatial acoustic experience of audio reproduction.
Using trained machines (such as neural networks) to separate stereo signals into multiple audio content categories, and perform crosstalk-free positioning and remixing them in space through a mixing module, using linear translation technology to maintain audio amplitude and phase information. Reduce artifact generation.
Effectively reduce or eliminate artifacts during separation, provide a more realistic spatial acoustic experience, and enhance the spatial positioning and distribution effects of audio reproduction.
Smart Images

Figure CN115226022B_ABST
Abstract
Description
background 1. Technical Field
[0002] Aspects of the invention relate to digital signal processing of audio, and in particular to audio content recorded in stereo and content-based separation and remixing.
[0003] 2. Description of Related Technology
[0004] Psychoacoustics is concerned with the human perception of sound. The sounds produced in a live performance interact acoustically with the environment, such as the walls and seats of a concert hall. After the sound waves travel through the air and before reaching the eardrum, they are filtered and delayed due to the size and shape of the head and ears. The signals received by the left and right ears differ slightly in level, phase, and time delay. The human brain processes the signals received from both auditory nerves simultaneously and derives spatial information about the location, distance, velocity, and environment of the sound source.
[0005] In a live performance recorded in stereo with two microphones, each microphone receives an audio signal with a time delay related to the distance between the audio source and the microphone. When the recorded stereo is played using a stereo reproduction system with two speakers, the original time delays and levels from the various sources to the microphones are reproduced as recorded. The time delays and levels provide the brain with a spatial sense of the original sound source. In addition, both the left and right ears receive audio from both the left and right speakers, a phenomenon known as channel cross-talk. However, if the same content is reproduced on headphones, the left channel is played only to the left ear, and the right channel is played only to the right ear, without reproducing channel cross-talk.
[0006] In a virtual binaural reproduction system using headphones with left and right channels, directionally dependent head-related transfer functions (HRTFs) can be used to simulate filtering and delay effects due to the size and shape of our heads and ears. Static and dynamic cues can be included to simulate the acoustics and movement of audio sources within a concert hall. Channel crosstalk can be restored. Taken together, these techniques can be used to virtually position the original audio sources in two-dimensional or three-dimensional space and provide a spatial acoustic experience to the user.
[0007] Brief Overview
[0008] Various computerized systems and methods are described herein, including a trained machine configured to input a stereo sound track and separate the stereo sound track into a plurality of N separated stereo audio signals, each of which is characterized by a plurality of N audio content categories. Basically, all stereo audio as input in the stereo sound track is included in the N separated stereo audio signals. A mixing module is configured to spatially position the N separated stereo audio signals symmetrically between left and right and without crosstalk to a plurality of output channels. The output channels include a corresponding mixture of one or more of the N separated stereo audio signals. The gain of the output channels is adjusted to the left and right binaural outputs to maintain the total level of the N separated stereo audio signals distributed on the output channels. The N audio content categories may include: (i) dialogue, (ii) music, and (iii) sound effects. The binaural reproduction system may be configured to present the output channels binaurally. The gains may be summed in phase within a previously determined threshold to suppress distortion generated during separation of the stereo audio track into N separate stereo audio signals. The binaural reproduction system may also be configured to spatially reposition one or more of the N separate stereo audio signals by linear panning. The sum of the audio amplitudes of the N separate stereo audio signals distributed on the output channels may be maintained. The trained machine may be configured to transform the input stereo audio track into an input time-frequency representation, process the time-frequency representation and output therefrom a plurality of time-frequency representations corresponding to the corresponding N separate stereo audio signals. For a time-frequency bin, the sum of the amplitudes of the output time-frequency representation is within a previously determined threshold of the amplitude of the input time-frequency representation. The trained machine may be configured to output a plurality of time-frequency representations of N-1 from the trained machine, and calculate the Nth time-frequency representation as a residual time-frequency representation by subtracting the sum of the amplitudes of the N-1 time-frequency representations for the time-frequency bin from the amplitude of the input time-frequency representation. The trained machine may be configured to prioritize at least one of the N audio content categories as a priority audio content category and process the priority audio content category serially by separating the stereo audio track into separate stereo audio signals of the priority audio content category before the other N-1 audio content categories. The priority audio content category may be dialogue. The trained machine may be configured to process the output time-frequency representation by extracting information for phase recovery from the input time-frequency representation.
[0009] Disclosed herein is a computer readable medium storing instructions for performing a computerized method as disclosed herein.
[0010] These, additional and / or other aspects and / or advantages of the invention are set forth in the detailed description which follows; may be inferred from the detailed description; and / or may be learned by practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The present invention is herein described, by way of example only, with reference to the accompanying drawings, in which:
[0012] Figure 1 shows a simplified schematic diagram of a system according to an embodiment of the present invention;
[0013] Figure 2 An embodiment of a separation module according to a feature of the present invention is shown, which is configured to separate an input stereo signal into N audio content categories or timbre categories (stems);
[0014] Figure 3 Another embodiment of a separation module according to a feature of the present invention is shown, which is configured to separate an input stereo signal into N audio content categories or timbre classes;
[0015] Figure 4 showing details of a trained machine according to features of the present invention;
[0016] Figure 5A An exemplary mapping of separated audio content categories (i.e., timbre categories) to virtual positions around a listener's head or virtual speakers according to features of the present invention is shown;
[0017] Figure 5B An example of spatial localization of separated audio content categories (i.e., timbre classification) according to features of the present invention is shown;
[0018] Figure 5C An example of an Envelopment by separate audio content categories (i.e., timbre classification) according to a feature of the present invention is shown; and
[0019] Figure 6 is a flow chart illustrating a method according to the present invention.
[0020] The above and / or other aspects will become apparent from the following detailed description when considered in conjunction with the accompanying drawings. Detailed Description
[0021] Reference will now be made in detail to the features of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. These features are described below to explain the present invention by referring to the drawings.
[0022] When mixing sounds for an animation, the audio content can be recorded as separate audio content categories, such as dialogue, music, and sound effects, also referred to herein as "timbre categories." Recording in timbre categories helps replace dialogue with foreign language versions, and also helps adapt the soundtrack to different reproduction systems, such as monaural, binaural, and surround sound systems.
[0023] However, conventional movies have one soundtrack that includes multiple audio content categories, such as dialogue, music, and sound effects, that were previously recorded together in stereo, for example, using two microphones.
[0024] Separating raw audio content into multiple timbre categories can be performed using one or more previously trained machines (e.g., neural networks). Representative references describing separation of raw audio content into multiple audio content categories using neural networks include:
[0025] Acidity Arie Nugraha, Antoine Liutkus, Emmanuel Vincent. Deep neural network based multichannel audio source separation. Audio Source Separation, Springer, pages 157-195, 2018, 978-3-319-73030-1.
[0026] S. Uhlich and M. Porcu and F. Giron and M. Enenkl and T. Kemp and N. Takahashi and Y. Mitsufuji, “Improving music source separation based on deep neural networks through data augmentation and network blending”. In 2017 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). IEEE, 2017.
[0027] The original audio content may not be completely separated, and the separation process may result in audible artifacts or distortion in the separated content. The separated audio content categories or timbre classifications can be virtually located in two-dimensional or three-dimensional space and remixed into multiple output channels. Multiple output channels can be input to an audio reproduction system to create a spatial sound experience. Features of the present invention relate to remixing the separated audio content categories and / or virtually locating the separated audio content categories in a manner that at least partially reduces or eliminates artifacts generated by an imperfect separation process.
[0028] Now refer to the accompanying drawings. Figure 1 , which shows a simplified schematic diagram of a system according to an embodiment of the present invention. A previously recorded input stereo signal 24 can be input into a separation block 10. The separation block 10 separates the input stereo 24 into a plurality of (e.g., N) audio content categories or timbre classifications. For example, the input stereo 24 can be a soundtrack of an animation, and the separation block 10 can separate the soundtrack 2 into N=3 audio content categories: (i) dialogue, (ii) music, and (iii) sound effects. The mixing block 12 receives the separated timbre classifications 1...N and is configured to remix and virtually position the separated timbre classifications 1...N. The positioning can be pre-set by the user, corresponding to a surround sound standard, such as 5.0, 7.1, or can be a free positioning in a surround plane or in a three-dimensional space. The mixing block 12 is configured to generate a multi-channel output 18, which can be stored on a binaural audio reproduction system 16 or otherwise played on the binaural audio reproduction system 16. Waves Nx TM Virtual Mix Room (Waves Audio) is an example of a binaural audio reproduction system 16. Waves Nx TM Designed to reproduce an audio mix in a spatial environment using a stereo or surround speaker configuration using conventional headphones comprising left and right physical on-ear or in-ear speakers.
[0029] Separates input stereo signal into multiple audio content categories
[0030] Now also refer to Figure 2 , which shows an embodiment 10A of a separation block 10 according to a feature of the present invention, which is configured to separate an input stereo signal 24 into N audio content categories or timbre classifications. The input stereo signal 24 may originate from a stereo animation audio track and may be input in parallel to a plurality of processors 20 / 1 to 20 / N-1 and a residual block 22 in number N-1. The processors 20 / 1 to 20 / N-1 are respectively configured to mask or filter the input stereo 24 to produce timbre classifications 1 to N-1.
[0031] Processors 20 / 1 to 20 / N-1 may be configured as trained machines, such as supervised machines that learn to output timbre classifications 1 ... N-1. Alternatively or additionally, unsupervised machine learning algorithms, such as principal component analysis, may be used. Block 22 may be configured to add together timbre classifications 1 to N-1 and may subtract the sum from the input stereo signal 24 to produce a residual output as timbre classification N, such that the sum of the audio signals from timbre classifications 1 ... N is substantially equal to the input stereo 24 within a previously determined threshold.
[0032] Taking N=3 timbre classifications as an example, processor 20 / 1 shields the input stereo 24 and outputs audio signal timbre classification 1, such as dialogue audio content. Processor 20 / 2 shields the input stereo 24 and outputs timbre classification 2, such as music audio content. Residual block 22 outputs timbre classification 3, which is basically all other sounds contained in the input stereo 24 that are not shielded and output by processors 20 / 1 and 20 / 2, such as sound effects. By using residual block 22, basically all sounds included in the original input stereo 24 are included in timbre classifications 1 to 3. According to the features of the present invention, timbre classifications 1 to N-1 can be calculated in the frequency domain and subtraction or comparison can be performed in the time domain in block 22 to output timbre classification N, thereby avoiding the final inverse transform.
[0033] Now also refer to Figure 3 , which shows another embodiment 10B of the separation block 10 according to a feature of the present invention, which is configured to separate an input stereo signal into N audio content categories or timbre classifications. The trained machine 30 / 1 inputs the input stereo 24 and masks the output timbre classification 1. The trained machine 30 / 1 is configured to output a residual 1 originally derived from the input stereo 24, which includes sounds in the input stereo 24 other than timbre classification 1. The residual 1 is input to the trained machine 30 / 2. The trained machine 30 / 2 is configured to mask the output timbre classification 2 from the residual 1 and output a residual 2, which includes sounds in the input stereo 24 other than timbre classifications 1 and 2. Similarly, the trained machine 30 / N-1 is configured to mask the output timbre classification N-1 from the residual N-2. The residual N-1 becomes the timbre classification N. As shown in the separation block 10B, all sounds included in the original input stereo 24 are included in the timbre categories 1 to N within the previously determined threshold. In addition, the separation block 10B is processed serially so that the most important timbre categories (such as dialogue) can be optimally masked with minimal distortion, and artifacts due to imperfect separation can tend to be integrated into subsequently masked timbre categories, such as into the timbre category 3 of sound effects.
[0034] Now also refer to Figure 4A block diagram of a machine 30 / 1 is shown schematically, by way of example, in detail of a trained machine 30 / 1 according to features of the invention. In block 40, the input stereo 24 may be resolved in the time domain and transformed into a frequency representation, such as a short-time Fourier transform (STFT). The short-time Fourier transform (STFT) 40 may be performed by sampling (e.g. 45 kHz) using an overlap-add method. A time-frequency representation 42 derived from the STFT, such as a real-valued spectrogram of the mixture, may be output or stored. A neural network initial layer 41 may clip the frequencies to a maximum frequency, such as 16 kHz, and scale the STFT to be more robust to variations in input level, such as by expressing the STFT relative to the mean amplitude and dividing by the standard deviation of the amplitude. For example, the initial layer 41 may include a fully connected layer, followed by a batch normalisation layer; and finally a non-linear layer, such as a tanh or sigmoid. The data output from the initial layer 41 can be input to a neural network core 43, which can include a recursive neural network, such as a three-layer long short-term memory (LSTM) network, which typically operates on time series data, in different configurations. Alternatively or additionally, the neural network core 43 can include a convolutional neural network (CNN) configured to receive two-dimensional data, such as a spectrogram in a time-frequency space. The output data from the neural network core 43 can be input to a final layer 45, which can include one or more hierarchical structures, the one or more hierarchical structures including a fully connected layer followed by a batch normalization layer. The rescaling performed in the initial layer 41 can be reversed. Finally, the transformed frequency data 44, such as the amplitude spectral densities corresponding to the timbre classification 1 (e.g., dialogue), is output from the nonlinear layer of block 45 (e.g., rectified linear unit, sigmoid or hyperbolic tangent (tanh)). However, in order to generate an estimate of the timbre classification 1 in the time domain, complex coefficients including phase information can be recovered.
[0035] Simple Wiener filtering or multi-channel Wiener filtering 47 can be used to estimate the complex coefficients of frequency data. Multi-channel Wiener filtering 47 is an iterative process using expectation maximization. The first estimate for the complex coefficient can be extracted from the STFT frequency bin 42 of the mixture and multiplied 46 with the corresponding frequency amplitude 44 output by the post-processing block 45. The Wiener filter 47 assumes that the complex STFT coefficients are independent zero-mean Gaussian random variables, and under these assumptions, the minimum mean square error of the source variance for each frequency is calculated. The output of the Wiener filter 47 and the STFT of the timbre classification 1 can be inversely transformed (block 48) to generate an estimate of the timbre classification 1 in the time domain. The trained machine 30 / 1 can calculate the output residual 1 in the frequency domain by subtracting the real-valued spectrogram 49 of the timbre classification 1 from the spectrogram 42 of the mixture as the output of the transform block 40. Residual 1 may be output to a trained machine 30 / 2, which may operate similarly to trained machine 30 / 1, however, since residual 1 is already in the frequency domain, transform 40 is redundant in trained machine 30 / 2. Residual 2 is output from trained machine 30 / 2 by subtracting STFT timbre classification 2 from residual 1 in the frequency domain.
[0036] Mixing and spatial localization of audio content categories
[0037] Reference again Figure 1 , the separation 10 of audio content categories may be limited so that, for example, all stereo audio originally recorded in a conventional animation stereo soundtrack is included in the separated audio content categories (i.e., timbre categories 1-3) (within a previously determined threshold). The timbre categories 1 ... N, (e.g., N = 3, dialogue, music, and sound effects) are mixed and positioned in a mixing block 12. The mixing block 12 may be configured to virtually map the separated N = 3 timbre categories: dialogue, music, and sound effects, to virtual locations around the listener's head.
[0038] Now also refer to Figure 5A , which shows an exemplary mapping of separate N=3 timbre categories: dialogue, music and sound effects to virtual positions or virtual speakers around the listener's head on a multi-channel output 18 by a mixing block 12. Five output channels are shown: center C, left L, right R, surround left SL and surround right SR. Timbre category 1 (e.g., dialogue) is shown mapped to the front center location C. Timbre category 2 (e.g., music) is shown mapped to the left front L and right front R locations shown shaded by the -45 degree line. Timbre category 3 (e.g., sound effects) is shown with cross-hatching as mapped to the left rear surround (SL) and right rear surround (SR) locations.
[0039] Now also refer to Figure 6, which shows a flow chart 60 of a computerized process of mixing into a plurality of channels 18 by a mixing module 12 to minimize artifacts caused by separation 10, according to features of the present invention. A stereo audio track is input (step 61) and separated (step 63) into N separate stereo audio signals characterized by N audio content categories. Separation (step 63) of the input stereo 24 into separate stereo audio signals of corresponding audio content categories may be constrained so that all audio of the original recording is included in the separated audio content categories. The mixing block 12 is configured to spatially position the N separate stereo audio signals between left and right into output channels.
[0040] The spatial localization between the left and right sides of the stereo may be performed symmetrically and without crosstalk between the left and right sides (step 65). In other words, the sound in the input stereo 24 originally recorded in the left channel is spatially localized (step 65) only in one or more left output channels (or center speakers), and similarly, the sound in the input stereo 24 originally recorded in the right channel is spatially localized in one or more right channels (or center speakers).
[0041] The gains of the output channels may be adjusted (step 67) to the left and right binaural outputs to maintain the summed level of the N separate stereo audio signals distributed across the output channels.
[0042] The output channels 18 may be presented binaurally (step 69) or alternatively reproduced in a stereo speaker system.
[0043] Reference now Figure 5B , which shows an example of spatial positioning of separated audio content categories (i.e., timbre categories) according to the features of the present invention. Timbre category 1 (e.g., dialogue) is shown as Figure 5A Shown is located at the front center virtual speaker C. Tone category 2 (Music L and R (Shaded - 45 lines)) with Figure 5A Compared to being relocated symmetrically to the left and right front at approximately ±30 degrees relative to the front centerline (FC) in the sagittal plane. Tone category 3 (sound effects (cross-hatching)) is relocated symmetrically between the left and right at approximately ±100 degrees relative to the front centerline. According to a feature of the invention, the spatial relocation can be performed by linear translation. For example, the spatial angle of spatial relocation of music R is shown. Music R gain G C is added to the center virtual speaker C, and the gain G of the right virtual speaker R R The gain G of the music R in the center virtual speaker C decreases linearly. C and the gain G of the music R in the right virtual speaker R RThe graph of is shown in the inset. The axis is the gain (ordinate) versus the spatial angle θ (abscissa) expressed in radians. The gain G of the music R in the center virtual loudspeaker C C and the gain G of the music R in the right virtual speaker R R Changes according to the following equation.
[0044]
[0045]
[0046] For the spatial angle, G C =1 / 3 and G R =2 / 3.
[0047] When linearly panned, the phases of the audio signals of the music R from the center virtual speaker C and from the right virtual speaker R are reconstructed so that for any spatial angle θ, the normalized effects of the two contributions to the music R add up to unity or close to unity. Furthermore, if the separation (block 10, step 63) is not perfect and the dialog peaks in the right channel are separated into the music R timbre category in the frequency representation, then the linear panning under the phase-preserving condition tends to at least partially restore the erroneous dialog peaks with the correct phase to the center virtual speaker that is presenting the dialog timbre category, tending to correct or suppress the distortion caused by the imperfect separation.
[0048] Reference now Figure 5C , which shows an example of enclosure of separate audio content categories (i.e., timbre categories) according to a feature of the present invention. Enclosure refers to the perception of sound around a listener and without a definable point source. Separate N=3 timbre categories: dialogue, music, and sound effects are displayed surrounding a listener's head at a wide angle. Timbre category 1 (e.g., dialogue) is displayed as generally coming from a forward direction at a wide angle. Timbre category 2 (e.g., music left and right) is displayed as coming from a wide angle as shown by the shading of the -45 degree line. Timbre category 3 (e.g., sound effects) is displayed as cross-hatched, surrounding the listener's head from behind at a wide angle.
[0049] The spatial envelopment between the left and right sides of the stereo is performed symmetrically and without crosstalk between the left and right sides (step 65). In other words, the sound in the input stereo 24 originally recorded in the left channel is spatially distributed only from the left output channel (or center speaker) (step 65), and similarly, the sound in the input stereo 24 originally recorded in the right channel is spatially distributed from one or more right channels (or center speakers). The phase is maintained so that the normalized gain in the output channels spatially distributed on the left side amounts to the unity gain of the left input stereo 24, and the normalized gain in the output channels spatially distributed on the right side amounts to the unity gain of the right input stereo 24.
[0050] Embodiments of the present invention may include general or special computer systems, which include various computer hardware components, which will be discussed in more detail below. Embodiments within the scope of the present invention also include computer-readable media for carrying or storing computer-executable instructions, computer-readable instructions or data structures. Such computer-readable media can be any available medium of temporary and / or non-temporary accessible to a general or special computer system. As an example and not limitation, such computer-readable media may include physical storage media, such as RAM, ROM, EPROM, flash disk, CD-ROM, or other optical disk storage devices, magnetic disk storage devices or other magnetic or solid-state storage devices, or any other medium that can be used to carry or store desired program code devices in the form of computer-executable instructions, computer-readable instructions or data structures and can be accessed by a general or special computer system.
[0051] In this specification and in the appended claims, "network" is defined as any architecture in which two or more computer systems can exchange data. The term "network" may include a wide area network, the Internet, a local area network, an intranet, a wireless network such as "Wi-Fi", a virtual private network, a mobile access network using an access point name (APN) and the Internet. The exchanged data may be in the form of an electrical signal that is meaningful to two or more computer systems. When data is transmitted or provided to a computer system or computer device via a network or another communication connection (or hardwired, wireless, or a combination of hardwired or wireless), the connection is appropriately regarded as a computer-readable medium. Therefore, any such connection is appropriately referred to as a computer-readable medium. The above combination should also be included in the scope of computer-readable media. Therefore, the computer-readable medium as disclosed herein may be temporary or non-temporary. Computer executable instructions include, for example, instructions and data that cause a general-purpose computer system or a special-purpose computer system to perform a certain function or group of functions.
[0052] The term "server" as used herein refers to a computer system including a processor, a data storage device, and a network adapter, which is generally configured to provide services over a computer network. A computer system that receives services provided by a server may be referred to as a "client" computer system.
[0053] The term "sound effects" as used herein refers to artificially created or enhanced sounds used to set a mood, simulate reality, or create an illusion in an animation. The term "sound effects" as used herein includes "foleys," which are sounds added to a production to provide a more realistic feel to an animation.
[0054] The term "source" or "audio source" as used herein refers to one or more sources of sound in a recording. Sources may include singers, actors / actresses, musical instruments, and sound effects, which may originate from a recording or be synthesized.
[0055] As used herein, the term "audio content category" refers to a classification of audio sources that may depend on the type of content, such as audio content categories where (i) dialogue, (ii) music, and (iii) sound effects are soundtracks suitable for animation. Other audio content categories may be considered based on type content, such as: strings, woodwinds, brass, and percussion instruments of a symphony orchestra. The terms "timbre classification" and "audio content category" are used interchangeably herein.
[0056] The term "spatial positioning" or "positioning" refers to the angular or spatial placement of one or more audio sources or timbre categories relative to a listener's head in two or three dimensions. The term "positioning" includes "surrounding," wherein audio sources are angularly and / or distancedly spread out to project sound toward a listener.
[0057] As used herein, the term "channel" or "output channel" refers to a mix of recorded audio sources or separated audio content categories that are presented for reproduction.
[0058] The term "binaural" as used herein refers to listening with two ears, like with headphones or with two speakers. The term "binaural rendering" or "binaural reproduction" refers to playing output channels in a positioning that provides a two-dimensional or three-dimensional spatial audio experience, for example.
[0059] As used herein, the term "maintain" means that the sum of the gains is equal to or close to a constant. For normalized gain, the constant is equal to or close to unity gain.
[0060] The term "stereo sound" as used herein refers to sound recorded using two left and right microphones and presented using at least two left and right output channels.
[0061] The term "crosstalk" as used herein refers to at least a portion of the sound recorded in the left microphone being presented to the right output channel, or similarly at least a portion of the sound recorded in the right microphone being presented in the left output channel.
[0062] As used herein, the term "symmetrically" refers to bilateral symmetry of positioning about a sagittal plane that divides the virtual listener's head into left and right mirrored halves.
[0063] As used herein in the context of audio signals, the terms "sum" or "summing" refer to combining signals that include corresponding frequencies and phases. For completely incoherent and / or uncorrelated audio waves, summing may refer to summing in terms of energy or power. For audio waves that are completely correlated in terms of phase and frequency, summing may refer to summing corresponding amplitudes.
[0064] As used herein, the term "panning" refers to adjusting the level according to a spatial angle, and in stereo, the levels of left and right output channels are adjusted simultaneously.
[0065] The terms "moving picture," "movie," "motion picture," and "film" are used interchangeably herein and refer to a multimedia product in which a soundtrack is synchronized with a video or moving picture.
[0066] Unless otherwise stated, the term "previously determined threshold" is implicitly included in the claims when appropriate, for example, "maintained" means "maintained within a previously determined threshold"; for example, "no crosstalk" means "no crosstalk within a previously determined threshold". Similarly, the terms "all", "substantially all", and "substantially all" refer to within a previously determined threshold.
[0067] The term "spectrogram" as used herein is a two-dimensional data structure in time-frequency space.
[0068] The indefinite articles "a", "an" as used in this document have the meaning of "one or more", i.e. for example, "a time-frequency bin", "a threshold" have the meaning of "one or more time-frequency bins" or "one or more thresholds".
[0069] All optional and preferred features and modifications of the described embodiments and the dependent claims are available in all aspects of the invention taught herein. In addition, each individual feature of the dependent claims and all optional and preferred features and modifications of the described embodiments are combinable and interchangeable with each other.
[0070] Although selected features of the present invention have been shown and described, it is to be understood that the invention is not limited to the described features.
Claims
1. A computerized method comprising: Input stereo track; separating the stereo audio track into a plurality of N separate stereo audio signals, the N separate stereo audio signals being characterized by a plurality of N audio content categories, respectively, while including all stereo audio that is input in the stereo audio track within a first previously determined threshold in the N separate stereo audio signals; binaurally rendering the N separated stereo audio signals into a plurality of output channels for use with headphones or stereo speakers, wherein audio amplitudes are summed in phase within a second previously determined threshold value to suppress distortion produced during said separation of the stereo audio track into the N separated stereo audio signals, wherein the output channels comprise respective mixtures of one or more of the N separated stereo audio signals; wherein the binaural rendering comprises listening with both ears to virtually spatially localize at least one of the N audio content categories, wherein sounds originally recorded in a left channel are rendered in one or more left output channels, and sounds originally recorded in a right channel are rendered in one or more right channels; and The gains of the output channels are adjusted into the left and right binaural outputs to maintain a total level of the N separate stereo audio signals distributed across the output channels.
2. The computerized method of claim 1, wherein: The N audio content categories include: (i) dialogue, (ii) music, and (iii) sound effects.
3. The computerized method of claim 1 , further comprising: One or more of the N separate stereo audio signals are spatially repositioned by panning.
4. The computerized method of claim 3, further comprising: in, The panning is linear, wherein the sum of the audio amplitudes of the N separate stereo audio signals distributed over the output channels is maintained.
5. The computerized method of claim 1 , further comprising: transforming the input stereo audio track into an input time-frequency representation; The time-frequency representation is processed by a trained machine and a plurality of time-frequency representations corresponding to respective N separated stereo audio signals are output therefrom, wherein, for a time-frequency bin, the sum of the amplitudes of the time-frequency representations is within a previously determined threshold of the amplitude of the input time-frequency representation.
6. The computerized method of claim 5, further comprising: Outputting a plurality of N-1 time-frequency representations from the trained machine; An Nth time-frequency representation is calculated by subtracting the sum of the magnitudes of the N-1 time-frequency representations with respect to the time-frequency bin from the magnitude of the input time-frequency representation as a residual time-frequency representation.
7. The computerized method of claim 6, further comprising: prioritizing at least one of the N audio content categories as a priority audio content category; and The at least one priority audio content category is processed serially by separating the stereo audio track into separate stereo audio signals of the priority audio content category prior to the other N-1 audio content categories.
8. The computerized method of claim 7, wherein: The priority audio content category is dialogue.
9. The computerized method of claim 5, further comprising: The time-frequency representation is processed by extracting information for phase recovery from the input time-frequency representation.
10. A non-transitory computer readable medium storing instructions that, when executed by a computer, perform the computerized method of claim 1.
11. A computerized system comprising: a trained machine configured to input a stereo audio track and separate the stereo audio track into a plurality of N separated stereo audio signals, the N separated stereo audio signals being characterized by a plurality of N audio content categories, respectively, wherein within a first previously determined threshold, all stereo audio that was input in the stereo audio track is included in the N separated stereo audio signals; A binaural reproduction system configured to binaurally render the N separate stereo audio signals into a plurality of output channels for use with headphones or stereo speakers, wherein audio amplitudes are summed in phase within a second previously determined threshold value to suppress distortion produced during said separation of the stereo audio track into the N separate stereo audio signals, wherein the output channels include respective mixtures of one or more of the N separate stereo audio signals, and the binaural reproduction system is configured to adjust gains of the output channels into left and right binaural outputs to maintain a summed level of the N separate stereo audio signals distributed across the output channels.
12. The computerized system of claim 11, wherein: The N audio content categories include: (i) dialogue, (ii) instrumental music, and (iii) sound effects.
13. The computerized system of claim 11, wherein: The binaural reproduction system is further configured to spatially reposition one or more of the N separate stereo audio signals by panning.
14. The computerized system of claim 13, wherein: The panning is linear, wherein the sum of the audio amplitudes of the N separate stereo audio signals distributed over the output channels is maintained.
15. The computerized system of claim 11, wherein: The trained machine is configured to: transforming the input stereo audio track into an input time-frequency representation; The time-frequency representation is processed and a plurality of time-frequency representations corresponding to respective N separated stereo audio signals are output therefrom, wherein for a time-frequency bin a sum of magnitudes of the time-frequency representations is within a previously determined threshold of the magnitude of the input time-frequency representation.
16. The computerized system of claim 15, wherein: The trained machine is configured to: outputting a plurality of N-1 time-frequency representations from the trained machine; and An Nth time-frequency representation is calculated by subtracting the sum of the magnitudes of the N-1 time-frequency representations with respect to the time-frequency bin from the magnitude of the input time-frequency representation as a residual time-frequency representation.
17. The computerized system of claim 16, wherein: The trained machine is configured to: prioritizing at least one of the N audio content categories as a priority audio content category; and The at least one priority audio content category is processed serially by separating the stereo audio track into separate stereo audio signals of the priority audio content category before the other N-1 audio content categories.
18. The computerized system of claim 17, wherein: The priority audio content category is dialogue.
19. The computerized system of claim 15, wherein: The trained machine is configured to: The time-frequency representation is processed by extracting information for phase recovery from the input time-frequency representation.
