Removing spatial artifacts from audio

Abstract

Audio applications determine the left amplitude of the audio source. LS k ) and the right amplitude of the audio source ( RS k Audio applications determine amplitude differences. D k Audio application computing D k time derivative d(D k ) Audio applications determine LS k and RS k The average value is used to obtain the mid-channel spectrum ( MCS k Audio applications for MCS k Normalization is performed to obtain the normalized value (R). k Audio applications will use d(D) K Divide by R k To obtain the confidence map, the audio application calculates the blending weights by scaling and cropping the confidence map. The audio application then combines the MCS (Multi-Segment Count). k Mixed weights and L t To obtain the modified left channel and combine the MCS k Mixed weights and R t To obtain the modified right channel.

Description

[0001] Cross-references to related applications

[0002] This application is an international application filed on August 12, 2024, entitled “REMOVALOF SPATIAL ARTIFACTS FROM AUDIO”, pursuant to 35 USC §119(e), the entire contents of which are hereby incorporated by reference. Background Technology

[0003] Audio processing, for various reasons such as source separation and noise removal, involves modifying the source audio. When the source audio has spatial characteristics (e.g., stereo audio including left and right channels or multi-channel audio), such processing may result in artifacts in the estimated or modified audio waveform.

[0004] Source separation models decompose complex audio signals composed of multiple overlapping audio sources into their constituent waveforms, including complex audio signals in stereo form (with separated left and right channels). The ability to independently isolate and reconstruct each audio source unlocks a wide range of possibilities for audio manipulation and analysis. For example, users can apply specific audio effects such as equalization or déreverberation to a single isolated audio source without affecting other elements of the mix; specific audio sources can be removed or suppressed, etc. Furthermore, separation algorithms can perform potentially detailed analysis of specific audio sources. Source separation models that perform audio source separation, particularly those employing mask-based separation in the time-frequency domain, have demonstrated significant efficiency and effectiveness. These source separation models operate by estimating a set of masks, one mask per audio source, which are then used to filter the complex audio signal in the time-frequency domain, effectively separating the different audio sources. This can then be applied to the time-frequency representation of the input audio and effectively separate the constituent sources into the input mixed recording.

[0005] However, despite their success, stereo sound separation models suffer from a significant limitation: introducing artifacts in the form of phase inconsistencies in multi-signal scenarios. Specifically, when a stereo sound separation model performs independent mask estimation on multiple signals representing the same audio—as in stereo separation where the left and right audio signals are portions of the same stereo audio—the resulting separation outputs may exhibit phase differences. These manifest as perceptible instabilities in the audio source's position relative to the listener's perceived location. When audio is played back after being processed using a conventional mask-based stereo separation model, it may exhibit an unnatural "jumping" effect, where one or more audio sources shift irregularly between the left and right signals, providing an undesirable effect of the source shifting irregularly from left to right or right to left from the listener's perspective. From the user's point of view, such random, temporary shifts in the spatial location of audio sources can be discordant and lead to a less than satisfactory experience.

[0006] The background description provided herein is for the purpose of presenting the general context of this disclosure. The work of the currently nominated inventors (to the extent described in this background section) and aspects of the specification that may not be considered prior art at the time of filing are neither expressly nor impliedly acknowledged as prior art of this disclosure. Summary of the Invention

[0007] A computer-implemented method modification includes having a corresponding left (…) for each audio source. L t ) channel and right ( R t Multiple channels n The audio stream from the audio source is separated from the original audio stream, where L t Audio channels and R t The audio channels are represented in time-frequency format. For each audio source ( k The method involves performing the following steps. This includes determining the left amplitude of the audio source. LS k ) and the right amplitude of the audio source ( RS k The method includes determining the LS. k and RS k The difference in amplitude between them D k The method includes calculating D. k time derivative d (D k ) The method further includes determining the LS. k and RS k The average value is used to obtain the mid-channel spectrum ( MCS k The method further includes the sum of the mid-channel spectrograms (MCS1 + MCS2 + ... + MCS) of each of the separated audio sources. n ) to MCS k Normalization is performed to obtain the normalized value (R). k The method further includes d(D) K Divide by R k A confidence map is obtained, wherein different regions of the confidence map are associated with corresponding probability values ​​indicating the likelihood that the region corresponds to a spatial artifact. The method further includes calculating mixing weights by scaling and clipping the confidence map. The method further includes combining MCS. k Mixed weights and L t To obtain the modified left channel and combine the MCS k Mixed weights and R t To obtain the modified right channel.

[0008] In some embodiments, the method further includes performing a modified left channel summation of two or more audio sources in the audio sources to obtain a left output channel, and performing a modified right channel summation of two or more audio sources in the audio sources to obtain a right output channel. In some embodiments, the method further includes performing an inverse short-time Fourier transform (STFT) on the left output channel to obtain a left playback channel, and performing an STFT on the right output channel to obtain a right playback channel, wherein the left and right playback channels are available for outputting audio via a speaker. In some embodiments, the method further includes receiving a command to erase a specific audio source, wherein two or more audio sources in the audio sources exclude the specific audio source from the left and right playback channels. In some embodiments, performing the summation includes applying a corresponding weight to each of the two or more audio sources.

[0009] In some embodiments, the method further includes separating the original audio stream from the video; providing the original audio stream as input to a source separation model; and using the source separation model to output an audio stream comprising multiple audio sources. In some embodiments, the method further includes determining the LS k and RS k Previously: Receive the raw audio stream, which includes the left ( L ) signal and right ( R The STFT is applied to the L and R signals respectively to obtain the left channel L signal. st and right channel R st Combination L st and R st Applying the source separation model to the combined Lst and R st To obtain the corresponding mask for each of the multiple audio sources; and to perform the corresponding mask with L st and R st The pointwise multiplication is performed to obtain the corresponding value for each audio source. L t and R t Multiple audio sources. In some embodiments, L is combined. st and R st Includes: calculating L st and R st The average value; and the magnitude of the average value.

[0010] A non-transitory computer-readable medium for modifying, including, for each audio source, a corresponding left ( L t ) channel and right ( R t Multiple channels n The audio stream from the audio source is separated from the original audio stream, where L t Audio channels and R t The audio channels are represented in a time-frequency manner, and instructions are stored on the non-transitory computer-readable medium that, when executed by one or more computers, cause one or more computers to perform operations, including, for each audio source ( k This operation includes: determining the left amplitude of the audio source ( ): . LS k ) and the right amplitude of the audio source ( RS k ); Determine LS k and RS k The difference in amplitude between them D k ); Calculate D k time derivative d(D k ) ; Determine LS k and RS k The average value is used to obtain the mid-channel spectrum ( MCS k ); based on the sum of the mid-channel spectrograms of each of the separated audio sources (MCS1 + MCS2 + … + MCS) n ) to MCS k Normalization is performed to obtain the normalized value (R). k ); d(D K Divide by R kTo obtain a confidence map, where different regions of the confidence map are associated with corresponding probability values ​​indicating the likelihood that the region corresponds to a spatial artifact; to calculate the mixed weights by scaling and clipping the confidence map; and to combine the MCS. k Mixed weights and L t To obtain the modified left channel and combine the MCS k Mixed weights and R t To obtain the modified right channel.

[0011] In some embodiments, the operation further includes performing a modified left channel summation of two or more audio sources in the audio sources to obtain a left output channel, and performing a modified right channel summation of two or more audio sources in the audio sources to obtain a right output channel. In some embodiments, the operation further includes performing an STFT on the left output channel to obtain a left playback channel, and performing an STFT on the right output channel to obtain a right playback channel, wherein the left playback channel and the right playback channel can be used to output audio via a speaker. In some embodiments, the operation further includes receiving a command to erase a specific audio source, wherein two or more audio sources in the audio sources exclude the specific audio source from the left playback channel and the right playback channel. In some embodiments, performing the summation includes applying a corresponding weight to each of the two or more audio sources.

[0012] In some embodiments, the operation further includes: separating the original audio stream from the video; providing the original audio stream as input to a source separation model; and using the source separation model to output an audio stream comprising multiple audio sources. In some embodiments, the operation further includes determining the LS k and RS k Previously: Receive the raw audio stream, which includes the left ( L ) signal and right ( R The STFT is applied to the L and R signals respectively to obtain the left channel L signal. st and right channel R st Combination L st and R st Applying the source separation model to the combined L st and R st To obtain the corresponding mask for each of the multiple audio sources; and to perform the corresponding mask with L st and R st The pointwise multiplication is performed to obtain the corresponding value for each audio source. L t and R t Multiple audio sources.

[0013] A computing device for modifying, for each audio source having a corresponding left ( L t ) channel and right ( R t Multiple channels n The audio stream from the audio source is separated from the original audio stream, where L t Audio channels and R t The audio channels are represented in a time-frequency manner. The computing device includes a processor; and a memory coupled to the processor that stores instructions thereon, which, when executed by the processor, cause the processor to perform operations, including, for each audio source ( k ): Determine the left amplitude of the audio source ( LS k ) and the right amplitude of the audio source ( RS k ); Determine LS k and RS k The difference in amplitude between them D k ); Calculate D k time derivative d(D k ) ; Determine LS k and RS k The average value is used to obtain the mid-channel spectrum ( MCS k ); based on the sum of the mid-channel spectrograms of each of the separated audio sources (MCS1 + MCS2 + … + MCS) n ) to MCS k Normalization is performed to obtain the normalized value (R). k ); d(D K Divide by R k To obtain a confidence map, where different regions of the confidence map are associated with corresponding probability values ​​indicating the region and the spatial artifact; to calculate the mixed weights by scaling and clipping the confidence map; and to combine the MCS. k Mixed weights and L t To obtain the modified left channel and combine the MCS k Mixed weights and R t To obtain the modified right channel.

[0014] In some embodiments, the operation further includes performing a modified left channel summation of two or more audio sources in the audio sources to obtain a left output channel, and performing a modified right channel summation of two or more audio sources in the audio sources to obtain a right output channel. In some embodiments, the operation further includes performing an STFT on the left output channel to obtain a left playback channel, and performing an STFT on the right output channel to obtain a right playback channel, wherein the left playback channel and the right playback channel can be used to output audio via a speaker. In some embodiments, the operation further includes receiving a command to erase a specific audio source, wherein two or more audio sources in the audio sources exclude the specific audio source from the left playback channel and the right playback channel. In some embodiments, performing the summation includes applying a corresponding weight to each of the two or more audio sources. Attached Figure Description

[0015] Figure 1 This is a block diagram of an example network environment for generating combined audio according to some embodiments described herein.

[0016] Figure 2 This is a block diagram of an example computing device for generating combined audio according to some embodiments described herein.

[0017] Figures 3A to 3B A flowchart is shown as an example method for removing spatial artifacts in audio using a spatial artifact filtering module, according to some embodiments described herein.

[0018] Figures 4 to 9 Different example spectrograms according to some embodiments described herein are shown.

[0019] Figure 10 A flowchart is shown of an example method for removing spatial artifacts in audio according to some embodiments described herein.

[0020] Figure 11 A flowchart is shown as another example method for removing spatial artifacts in audio, according to some embodiments described herein. Detailed Implementation

[0021] The technique described in this paper utilizes the temporal dynamics of the amplitude difference between stereo signals to automatically identify and suppress artifacts after applying an initial source separation model to the input middle channel mixed signal and performing initial source separation on the middle channel signal. This technique advantageously operates directly on the pre-computed Short-Time Fourier Transform (STFT) from the source separation model, making it computationally efficient. By automatically suppressing artifacts, this technique enhances the perceptual quality of the output audio obtained by combining the separated audio sources.

[0022] Example Environment

[0023] Figure 1 A block diagram illustrating an example environment 100 for generating combined audio is shown. In some embodiments, environment 100 includes a media server 101 coupled to a network 105 and a user device 115. A user 125 may be associated with user device 115. In some embodiments, environment 100 may include Figure 1 Other servers or devices not shown in the diagram. Figure 1 In the remaining figures, letters following the reference numerals, such as "103a," indicate a reference to an element having that particular reference numeral. Reference numerals in the text without following letters, such as "103," indicate a general reference to an embodiment of the element carrying that reference numeral.

[0024] Media server 101 may include a processor, memory, and network communication hardware. In some embodiments, media server 101 is a hardware server. Media server 101 is communicatively coupled to network 105 via signal line 102. Signal line 102 may be a wired connection, such as Ethernet, coaxial cable, fiber optic cable, etc., or a wireless connection, such as Wi-Fi®, Bluetooth®, or other wireless technologies. In some embodiments, media server 101 sends data to and receives data from user device 115 via network 105. Media server 101 may include audio application 103a and database 199.

[0025] Database 199 can store machine learning models, training datasets, raw videos, enhanced videos, etc. Database 199 can also store social network data associated with user 125, user preferences of user 125, etc.

[0026] User device 115 is a computing device that includes memory coupled to a hardware processor. For example, user device 115 may include a mobile device, tablet computer, laptop computer, desktop computer, mobile phone, wearable device, head-mounted display, mobile email device, portable game player, portable music player, or another electronic device capable of accessing network 105.

[0027] In the illustrated implementation, user device 115 is coupled to network 105 via signal line 108. Signal line 108 can be a wired connection, such as Ethernet, coaxial cable, fiber optic cable, etc., or a wireless connection, such as Wi-Fi®, Bluetooth®, or other wireless technologies. Figure 1 User device 115 in the example is used only. Although Figure 1 A user device 115 is shown, but this disclosure applies to system architectures having one or more user devices 115.

[0028] Audio application 103 may be stored on media server 101 or user device 115. In some embodiments, the operations described herein are performed on media server 101 or user device 115. In some embodiments, some operations may be performed on media server 101 and some operations may be performed on user device 115. In some embodiments, audio application 103b stored on user device 115 receives updates from audio application 103a stored on media server 101.

[0029] The execution of operations is based on user settings. For example, user 125 may specify the following setting: the operation should be performed on user device 115 instead of media server 101. Under such a setting, the operations described herein are performed entirely on user device 115, and no operation is performed on media server 101. Furthermore, user 125 may specify that the user's video and / or other data is stored locally only on user device 115 and not on media server 101. Under such a setting, no user data is transferred to or stored on media server 101. The transfer of user data to media server 101, any temporary or permanent storage of such data by media server 101, and the execution of operations on such data by media server 101 are only performed if the user has consented to the transfer, storage, and operation execution via media server 101. Users are provided with the option to change the settings at any time, such as enabling or disabling the use of media server 101.

[0030] Machine learning models (e.g., neural networks or other types of models) are stored and utilized locally on user device 115 with specific user permissions for use in one or more operations. Server-side models are used only with user permission. Furthermore, trained models can be provided for use on user device 115. During such use, on-device training of the model can be performed with permission from user 125. Updated model parameters can be sent to media server 101 with permission from user 125, for example, to enable federated learning. Model parameters do not include any user data.

[0031] In some embodiments, audio application 103 may be implemented using hardware, including a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, audio application 103a may be implemented using a combination of hardware and software.

[0032] Audio application 103 modification includes having a corresponding left (…) for each audio source.L t ) channel and right ( R t Multiple channels n The audio stream from the audio source is separated from the original audio stream, where L t Audio channels and R t The audio channel is represented in time-frequency format. Audio application 103 determines the left amplitude of the audio source (...). LS k ) and the right amplitude of the audio source ( RS k Audio application 103 determines LS. k and RS k The difference in amplitude between them D k Audio application 103 calculation D k time derivative d(D k ) Audio application 103 determines LS k and RS k The average value is used to obtain the mid-channel spectrum ( MCS k Audio application 103 is based on the sum of the mid-channel spectrograms of each separate audio source in the separate audio sources (MCS1+ MCS2+ … + MCS). n ) to MCS k Normalization is performed to obtain the normalized value (R). k Audio application 103 will d(D) K Divide by R k A confidence map is obtained, where different regions of the confidence map are associated with corresponding probability values ​​indicating the likelihood that the region corresponds to a spatial artifact. Audio application 103 calculates the mixing weights by scaling and clipping the confidence map. Audio application 103 combines the MCS. k Mixed weights and L t To obtain the modified left channel, and combined MCS k Mixed weights and R t To obtain the modified right channel.

[0033] Example computing device

[0034] Figure 2 This is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. The computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is an audio server 101 for implementing audio application 103a. In another example, the computing device 200 is a user device 115.

[0035] In some embodiments, the computing device 200 includes a processor 235, a memory 237, an input / output (I / O) interface 239, a microphone 241, a speaker 243, a display 245, a camera 247, and a storage device 249, all coupled via a bus 218. The processor 235 may be coupled to the bus 218 via signal line 222, the memory 237 may be coupled to the bus 218 via signal line 224, the I / O interface 239 may be coupled to the bus 218 via signal line 226, the microphone 241 may be coupled to the bus 218 via signal line 228, the speaker 243 may be coupled to the bus 218 via signal line 230, the display 245 may be coupled to the bus 218 via signal line 232, the camera 247 may be coupled to the bus 218 via signal line 234, and the storage device 249 may be coupled to the bus 218 via signal line 236.

[0036] Processor 235 may be one or more processors and / or processing circuits for executing program code and controlling the basic operations of computing device 200. "Processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may include systems having: a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit system for implementing functionality, a dedicated processor for implementing neural network-based processing, neural circuits, a processor optimized for matrix computations (e.g., matrix multiplication), or other systems. In some embodiments, processor 235 may include one or more coprocessors implementing neural network processing. In some embodiments, processor 235 may be a processor that processes data to produce probabilistic outputs; for example, the output produced by processor 235 may be imprecise or may be accurate within a range from the expected output. Processing is not required to be geographically restricted or time-constrained. For example, a processor may perform its functions in real-time, offline, in batch mode, etc. The different parts of the process can be executed at different times and in different locations by different (or the same) processing systems. The computer can be any processor that communicates with memory.

[0037] Memory 237 is typically provided in computing device 200 for access by processor 235 and can be any suitable processor-readable storage medium suitable for storing instructions for execution by processor or multiple processors and located separately from and / or integrated with processor 235, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. Memory 237 may store software operated on computing device 200 by processor 235, including audio application 103.

[0038] The memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, video library applications, video management applications, video zone applications, communication applications, web hosting engines or applications, media sharing applications, etc. One or more methods disclosed herein can operate in various environments and platforms, for example, as a standalone computer program that can run on any type of computing device, as a web application with web pages, as a mobile application (“app”) running on a mobile computing device, etc.

[0039] Application data 266 may be data generated by other applications 264 or the hardware of computing device 200. For example, application data 266 may include video used by a video library application and user actions identified by other applications 264 (e.g., a social networking application).

[0040] I / O interface 239 provides functionality to enable computing device 200 to interface with other systems and devices. Interface devices may be included as part of computing device 200, or they may be separate and communicate with computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 249), and input / output devices may communicate via I / O interface 239. In some embodiments, I / O interface 239 may be connected to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, scanner, sensor, etc.) and / or output devices (display device, speaker device, printer, monitor, etc.).

[0041] Microphone 241 may include hardware for detecting sound. For example, microphone 241 may be used as part of user device 115 to detect ambient noise, human speech, music, etc. In some embodiments, microphone 241 may include multiple audio sensors (e.g., two audio sensors, four audio sensors, or any number of audio sensors). In some embodiments, the microphone 241 sensors may detect audio from a mono clip-on microphone 241 for a cameraman's speech, a stereo ambient microphone 241 for a natural sound environment, and a mono directional microphone 241 for a specific person's speech. The audio detected by the individual audio sensors of microphone 241 may be combined to obtain an audio signal.

[0042] The speaker 243 may include hardware for generating an audible audio signal. In some embodiments, the speaker 243 includes an amplifier for amplifying certain channels, frequencies, etc. In some embodiments, the amplifier performs automatic gain control to ensure a consistent output signal amplitude regardless of variations in the input signal amplitude. In some embodiments, the device may also support auxiliary audio playback, such as via headphones (wired or wireless), remote speakers (e.g., connected via Bluetooth or other protocols), etc.

[0043] Display 245 includes a user interface for displaying content—such as images, videos, and / or output applications as described herein—and hardware for receiving touch (or gesture) input from a user. For example, display 245 may be used to display a user interface that includes user preferences for audio types. Display 245 may include any suitable display device, such as a liquid crystal display (LCD), a light-emitting diode (LED) or plasma display, a cathode ray tube (CRT), a television, a monitor, a touchscreen, a 3D display, or other visual display device. For example, display 245 may be a flat panel display mounted on a mobile device, multiple displays embedded in an eyeglass-shaped or head-mounted device, or a monitor screen for a computer device.

[0044] Camera 247 can be any type of image capture device capable of capturing images and / or video. In some embodiments, camera 247 captures images or video through I / O interface 239 for transmission to audio application 103.

[0045] Storage device 249 stores data related to audio application 103. For example, storage device 249 may store a training dataset including training data (such as multiple labeled audio files, source separation models, raw audio files, audio files with spatial artifacts removed, etc.).

[0046] Figure 2An example audio application 103 is illustrated, which includes a user interface module 202, a processing module 204, a source separation module 206, a spatial artifact filtering module 208, and an output module 210. In some embodiments, each of the components includes a set of instructions executable by a processor 235 to perform the steps discussed in more detail below. In some embodiments, each of the components is stored in a memory 237 of a computing device 200 and may be accessible and executable by the processor 235. In some embodiments, one or more of modules 202 through 210 may be implemented in dedicated hardware such as a digital signal processor, an audio processor, an FPGA or application-specific integrated circuit (ASIC), a machine learning processor, etc.

[0047] User interface module 202 generates graphical data for displaying the user interface. In some embodiments, the user interface displays options for capturing video and / or audio. In some embodiments, user interface module 202 generates a user interface that includes options for the user to specify user preferences. User preferences may include options for consenting to the use of the audio enhancement techniques described herein to process user-created videos, sending video and / or audio to a server for processing, etc. User preferences may also include options for specifying preferences regarding the type of auditory object, such as options for excluding specific audio sources. For example, audio application 103 may identify four audio sources in an audio stream: baby, mother, pet, and car noise. The user can select an option to exclude car noise from the audio stream.

[0048] Processing module 204 processes audio. In some embodiments, processing module 204 receives video and separates the raw audio stream from the video. In some embodiments, processing module 204 receives the audio stream without referencing the video. For example, processing module 204 may receive audio from speaker 243.

[0049] The raw audio stream includes the left ( L ) signal and right ( R The left and right signals correspond to the left and right speakers, respectively, and are used to create stereo sound. Each of the left and right signals is a time-domain waveform.

[0050] Processing module 204 applies short-time Fourier transform (STFT) to the L signal and R signal respectively to obtain the left channel ( L st ) and right channel ( R st The STFT is calculated by taking the discrete Fourier transform of a small moving window that spans the window duration, based on the waveforms of the left and right signals, respectively.

[0051] The STFT matrix is ​​a two-dimensional complex-valued matrix where the position of each entry in the STFT determines its time and frequency. Frequency is represented by the y-axis of the spectrum, and time is represented by the x-axis. A specific entry in the matrix is ​​called a time-frequency bin. Each time-frequency bin represents the amplitude of the audio signal at a specific time and frequency. The absolute value of the time-frequency bin, i.e., at time (…), represents the amplitude of the audio signal at that specific time and frequency. t ) and frequency ( f |X(t, f)| at time () is determined at time () t From frequency () f The amount of energy heard. Each time-frequency bin contains both amplitude and phase components. The STFT matrix is ​​used to separate the audio stream into audio sources and to manipulate those sources. After manipulating the audio sources, the amplitude and phase components of the STFT matrix are used to invert the STFT matrix back into a waveform, making the audio source intelligible to the human ear.

[0052] Processing module 204 combines the left and right channels to form what is called the combined L. st and R st The processing module 204 combines the left and right channels by calculating the average values ​​of the left and right channels and the amplitude of the average values ​​of the left and right channels. The processing module 204 then combines the combined L... st and R st Provided to source separation module 206.

[0053] Source separation model identifiers constitute multiple ( ) of the original input mixed sound n The estimated audio sources are identified as the combined L. st and R st (in) k The sum of 1) independent real-value source signals. For example, if the audio is a musical performance, then the audio can combine the L... st and R st The audio sources are separated into guitar, piano, voice, and drum audio sources. The source separation module 206 outputs a mask for each audio source within the audio sources.

[0054] In some embodiments, the source separation model is a machine learning model. The source separation model trained by the source separation module 206 may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep learning neural network that implements multiple layers (e.g., "hidden layers" between the input layer and the output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, uses one or more neural network layers to process each tile individually and aggregates the results of the processing from each tile), a sequence-to-sequence neural network (e.g., a network that takes sequence data—such as words in a sentence, frames in a video, etc.—as input and produces a sequence of results as output), etc.

[0055] The model form or structure can specify the connectivity between individual nodes and the organization of nodes into layers. For example, nodes in the first layer (e.g., the input layer) can receive data as input data or application data. For example, when a trained model is used for analysis, such as audio, this data may include, for example, one or more waveforms or STFTs for each node. Subsequent intermediate layers can receive the outputs of nodes in previous layers as input, depending on the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. The final layer (e.g., the output layer) produces the output of the machine learning model. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0056] In some embodiments, source separation module 206 may include multiple trained source separation models. One or more audio separation models in the source separation models may include multiple nodes arranged in layers according to a model structure or form. In some embodiments, a node may be a memoryless computation node, for example, configured to process the input of a unit to produce the output of a unit. Computations performed by a node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to produce a node output. In some embodiments, computations performed by a node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such computations may include operations such as matrix multiplication. In some embodiments, computations performed by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, individual processing units using a graphics processing unit (GPU), or a dedicated neural circuit system. In some embodiments, a node may include memory, for example, being able to store and use one or more earlier inputs while processing subsequent inputs. For example, a node with memory may include a Long Short-Term Memory (LSTM) node. LSTM nodes can use memory to maintain the "state" of the permitted node, which acts like a finite state machine (FSM).

[0057] In some embodiments, the trained model may include embeddings or weights for individual nodes. For example, the model may be started as multiple nodes organized into layers, as specified by model form or structure. Upon initialization, corresponding weights may be applied to the connections between each pair of nodes connected in model form—e.g., nodes in consecutive layers of a neural network. For example, the corresponding weights may be randomly assigned or initialized to default values. The source separation model can then be trained, for example, using training data to produce results.

[0058] Training may include applying supervised learning techniques. In supervised learning, training data may include multiple inputs (e.g., multiple training video clips) and corresponding ground truth outputs for each input (e.g., ground truth audio channels from a specific audio source in an audio clip). The weights are automatically adjusted based on a comparison between the model's outputs (e.g., predicted channels) and the ground truth outputs (e.g., ground truth channels), for example, to increase the probability that the model will produce ground truth channels.

[0059] In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In some embodiments, the trained source separation model may include, for example, an initial set of weights downloaded from a server providing weights. In various embodiments, the trained source separation model includes a set of weights or embeddings corresponding to the model structure. In embodiments where data is omitted, the source separation module 206 may generate a trained source separation model based on prior training performed, for example, by the developer of the source separation module 206, a third party, etc.

[0060] In some embodiments, where the source separation model includes a convolutional neural network trained using supervised learning, training the source separation model may include obtaining predicted channels based on the training clip for each training clip. The source separation model may compute a loss value based on a comparison of the predicted channels of the audio clip and the true channels (included in the training data). The source separation model may update the weights of one or more nodes of the convolutional neural network based on the loss value (e.g., in such a way that after adjustment and running another training epoch, the loss value is reduced until the loss value is below a threshold). In some embodiments, the source separation model includes a learnable convolutional encoder and decoder layer with a temporal convolutional network masking network.

[0061] Processing module 204 receives the mask from source separation module 206 and performs corresponding mask and L... st and R st Pointwise multiplication is performed to obtain the corresponding left ( ) for each audio source. L t ) channel and right ( R t Multiple audio sources for each channel. Therefore, if four audio sources are identified and each audio source is associated with a left channel and a right channel, point-by-point multiplication produces eight channels.

[0062] The audio source is provided to the spatial artifact filtering module 208. Although the spatial artifact filtering module 208 receives audio sources processed using STFT, the filtering process can also be applied to audio represented by a model different from STFT. The spatial artifact filtering module 208 utilizes the temporal dynamics of the amplitude difference between stereo channels to identify and suppress audio artifacts.

[0063] For each audio source in the audio source, the spatial artifact filtering module 208 operates as follows. The spatial artifact filtering module 208 determines the left amplitude value of the audio source ( LS k ) and the right amplitude of the audio source ( RS k Spatial artifact filtering module 208 determines LS. k and RS k The difference in amplitude between themD k Spatial artifact filtering module 208 calculates LS. k and RS k The difference in amplitude between them D k The time derivative of ), referred to in this paper as d(D k ) Spatial artifact filtering module 208 determines LS k and RS k The average value is used to obtain the mid-channel spectrum ( MCS k The spatial artifact filtering module 208 filters based on the sum of the mid-channel spectrograms of each of the separate audio sources (MCS1 + MCS2 + … + MCS). n ) to MCS k Normalization is performed to obtain the normalized value (R). k To calculate the relative source energy.

[0064] Spatial artifact filtering module 208 will d(D K Divide by R k A confidence map is obtained. The confidence map is divided into different regions, each associated with a corresponding probability value indicating the likelihood that the region corresponds to a spatial artifact and not to an audio source. When a region may be caused by a spatial artifact rather than an audio source, the spatial artifact filtering module 208 replaces the region with an intermediate channel estimate.

[0065] The spatial artifact filtering module 208 calculates the blending weights by scaling and clipping the confidence map. The blending weights determine the amount of mid-channel information to be blended with the original channel signal. The blending weights are referred to below as the alpha array. The spatial artifact filtering module 208 combines the MCS. k Mixed weights and L t To obtain the modified left channel and combine the MCS k Mixed weights and R t To obtain the modified right channel.

[0066] For each audio source, output module 210 receives the modified left channel and modified right channel from spatial artifact filtering module 208. Output module 210 performs summation of the modified left channels of two or more audio sources to obtain the left output channel, and performs summation of the modified right channels of two or more audio sources to obtain the right output channel.

[0067] Output module 210 performs inverse STFT on the left and right output channels to obtain source stereo waveforms (i.e., source left waveform and source right waveform). In some embodiments, output module 210 merges some sources by summing the waveforms of some sources to reduce the number of tracks. Merging avoids presenting near-silent or redundant sources in isolation during playback heard by the user when the audio source is modified in the user interface.

[0068] In some embodiments, a user can request the erasure of a specific audio source, such as an audio source corresponding to construction noise. The user can specify the request to erase an audio source via a user interface. In response to a user request to erase a specific audio source, the output module 210 excludes the specific audio source from two or more audio sources.

[0069] In some embodiments, a user can request to increase or decrease the audio level of different audio sources. For example, if the audio sources are speech, music, and nature, the user may want to increase the volume of the speech and decrease the volume of the music and nature audio sources.

[0070] In some embodiments, output module 210 applies corresponding weights to two or more audio sources. For example, desired audio sources (e.g., human speech, musical instruments, etc.) are associated with higher weights than distracting audio sources (e.g., background noise such as traffic, construction, crowd noise, background hum, etc.; temporary loud sounds such as car horns, etc.). Output module 210 may determine the corresponding weights based on the type of preferred audio source specified by the user. In some embodiments, a weight of 0 for an audio source is associated with an erased audio source.

[0071] Output module 210 sums the source stereo waveforms based on weights to form the left and right playback channels. In some embodiments, the left and right playback channels are passed through limiters to prevent audio clipping. The result from the limiters can be provided to speaker 243 for output. In some embodiments, output module 210 retains the original phase information from the input STFT.

[0072] Example Method

[0073] Figures 3A to 3B A flowchart of an example method 300 for removing spatial artifacts in audio using a spatial artifact filtering module, according to some embodiments described herein, is shown. The method begins by receiving an audio stream comprising an L signal 302 and an R signal 304. An STFT 306 is applied to the L signal 302 and an STFT 308 is applied to the R signal 304 to obtain the left channel L, respectively. st and right channel R stThe left and right channels are averaged 310, and the amplitude 312 of the average value 310 is determined. This is called the middle channel. The middle channel is provided as input to the source separation model 314.

[0074] The source separation model 314 outputs a mask for each audio source present in the average channel. A pointwise multiplication of the corresponding mask with the left channel 316 and the right channel 318 is performed to obtain multiple audio sources with corresponding left and right channels for each audio source. The audio sources are then provided to the spatial artifact filtering module 319.

[0075] For each audio source provided to the spatial artifact filtering module 319, the following operations are performed: The amplitude of the left channel 320 and the amplitude of the right channel 322 of the audio source are determined. The amplitude difference 324 between the absolute values ​​of the left and right channel amplitudes is determined. In some embodiments, instead of calculating the amplitudes of the left and right channels, the square of the left channel amplitude and the square of the left channel amplitude are calculated. For example, a finite difference in time center is used to determine the time derivative 326 of the amplitude difference. In some embodiments, the time derivative 326 is calculated using the following equation: Time gradient increment magnitude = gradient magnitude difference / (magnitude difference + 1e) -9 Equation 1 The average value 328 of the left and right amplitudes of the audio sources is determined to obtain a mid-channel spectrogram. For example, the equation used to calculate the mid-channel spectrogram could be the absolute value of the sum of the left and right amplitudes divided by 2. The mid-channel spectrograms for all audio sources are summed and normalized 330 to obtain the relative source energy for each corresponding audio source. The time derivative is divided by the relative source energy 332 to create a spatial (time-frequency) artifact confidence map. In some embodiments, the spatial artifact confidence map is calculated using the following equation: Spatial artifact confidence = temporal gradient increment magnitude / relative source energy (Equation 2) Gradient magnitude difference (Sum of the spectrum of all channels from all sources) / [(Amplitude difference + 1e] -9 ) [Middle Channel Spectrum Diagram] Equation 3 The mixed weights are calculated by scaling and cropping portions of the 334 confidence graph. In some embodiments, the mixed weights are calculated using the following equation: α = clip(spatial artifact confidence / (spatial artifact tolerance + 1e)) -9 Equation 4 (0, 1) The middle channel spectrogram, mixing weights, and left channel are combined in 336 to obtain the left output STFT (also known as the modified left channel). The middle channel spectrogram, mixing weights, and right channel are combined in 338 to obtain the right output STFT (also known as the modified right channel). Spatial artifacts are significantly reduced or eliminated in the modified left and modified right channels compared to the output from source separation model 314. The complex phase of the output STFT is the same as the phase of the input stereo STFT. In some embodiments, the amplitude is replaced by the following equation: New amplitude = (1 – α) Initial channel amplitude + α Equation 5 for middle channel amplitude In some embodiments, the left output STFT and right output STFT for each audio source in the audio source are combined. In some embodiments, except for those corresponding audio sources that are requested to be erased by the user, the left output STFT and right STFT for each audio source in the audio source are combined. Inverse STFT 340 is applied to the combined left output STFT to obtain an artifact-free left output. Inverse STFT 342 is applied to the combined right output STFT to obtain an artifact-free right output. Once inverse STFTs 340 and 342 are applied, the output is audible to the user.

[0076] The process described above is in Figures 4 to 9 Example spectrograms are shown. In each spectrogram, the white vertical bands represent speech. Ridges within the white vertical bands represent artifacts, such as background noise. The goal of the filtering process is to reduce ridges and smooth their appearance.

[0077] Figure 4 This is an example spectrogram 400 that includes the absolute values ​​of spatial artifact confidence. The spatial confidence is a 2D array with the same dimensions as the STFT. The spatial confidence represents the confidence for each time-frequency unit in whether an artifact is present, where larger values ​​(darker in the figure) indicate a greater confidence in the artifact.

[0078] Figure 5 Example spectrogram 500 includes the robust derivative of the absolute value of the confidence score for spatial artifacts and the maximum value of the one-sided difference. In some embodiments, both the forward and backward one-sided differences of the spectrogram are calculated over time, and the side including the largest amplitude is selected. The robust derivative doubles the temporal resolution while still providing a centered estimate, resulting in a clearer confidence plot.

[0079] Figure 6Example spectrogram 600 is another embodiment for performing spatial artifact filtering, which uses the absolute value of the spatial artifact confidence, the maximum value of the one-sided difference, and the power. Instead of calculating the amplitude difference between the absolute values ​​of the amplitudes of the left and right channels, filtering is performed on the power, where the formula f(x) = |x|^2 is smooth at x=0, while |x| is not, which can improve the accuracy of derivative estimation.

[0080] Figure 7 It shows the relationship with Figure 6 The same example spectrogram 700, and before and after examples of how spatial artifacts appear after filtering is performed.

[0081] Figure 8 An example spectrogram 800 is shown, using the absolute value of the spatial artifact confidence, the maximum value of the one-sided difference, power, and smoothing. In this example, the example spectrogram 800 undergoes a Gaussian smoothing operator that operates as a convolution operator in both time and frequency. In some embodiments, both the numerator and denominator are smoothed. Smoothing can be performed by a variety of different operators, including non-convolution operators. For example, since the speech spectrum tends to decrease rapidly in energies above 4 kHz, there may be practical advantages to smoothing for a stronger / more blurred effect in higher frequency units to address lower signal-to-noise ratios.

[0082] Figure 9 Example spectrum 900 of the α array after the spatial artifact process has been applied is shown, where the spatial artifact tolerance = 50. (Compared to...) Figure 8 compared to, Figure 9 The results of spatial artifact removal are shown with a larger spatial artifact tolerance threshold to illustrate the effect of this parameter. Due to the change in spatial artifact tolerance, more of the original signal is retained in the final output.

[0083] Example Flowchart

[0084] Figure 10 An example flowchart of a method 1000 for removing spatial artifacts in audio according to some embodiments described herein is shown. Method 1000 includes an audio stream comprising a corresponding left (…) for each audio source. L t ) channel and right ( R t Multiple channels n An audio source, which is separate from the original audio stream, where L t Audio channels and R t The vocal tract is represented in time-frequency format.

[0085] Method 1000 can be derived from Figure 2The method is executed by the computing device 200. In some embodiments, the method 1000 is performed by... Figure 1 The user device 115 executes this.

[0086] Figure 10 Method 1000 may begin at box 1002. In box 1002, the raw audio stream is received, the raw audio stream including the left (…). L ) signal and right ( R (Signal). In some embodiments, the original audio stream is separated from the video. Box 1002 may be followed by box 1004.

[0087] In box 1004, the STFT is applied to the L signal and R signal respectively to obtain the left channel L. st and right channel R st Box 1004 can be followed by box 1006.

[0088] In box 1006, group L st and R st In some embodiments, the combination L st and R st Includes: calculating L st and R st The average value; and the magnitude of the average value. Box 1006 can be followed by box 1008.

[0089] In box 1008, the source separation model is applied to the combined L st and R st This is to obtain the corresponding mask for each of the multiple audio sources. Box 1008 can be followed by box 1010.

[0090] In box 1010, apply the corresponding mask and L. st and R st Pointwise multiplication is used to obtain the corresponding value for each audio source. L t and R t Multiple audio sources. Box 1010 can be followed by box 1012.

[0091] In box 1012, select the sound source (k) and the corresponding mask. Box 1012 can be followed by box 1014.

[0092] In box 1014, spatial artifact removal is performed on the selected sound source and its corresponding mask. The spatial artifact removal process may include... Figure 11 Method 1100 is described in the text. Box 1014 can be followed by box 1016.

[0093] In box 1016, determine if there are additional audio sources. If there are additional audio sources, box 1016 can be followed by box 1012. If there are no additional audio sources, box 1016 can be followed by box 1018.

[0094] In box 1018, a weighted summation is performed for each left output channel and each right output channel. Box 1018 can be followed by box 1020.

[0095] In box 1020, inverse STFT is applied to obtain the left playback channel and the right playback channel.

[0096] Figure 11 An example flowchart of another method 1100 for removing spatial artifacts in audio according to some embodiments described herein is shown. Method 1200 includes an audio stream comprising a corresponding left (…) for each audio source. L t ) channel and right ( R t Multiple channels n An audio source, which is separate from the original audio stream, where L t Audio channels and R t The vocal tract is represented in time-frequency format. Method 1200 can be derived from... Figure 2 The method 1100 is executed by the computing device 200. In some embodiments, the method 1100 is performed by... Figure 1 User device 115 executes.

[0097] Figure 11 Method 1100 can begin at box 1102. In box 1102, determine the left amplitude value of the audio source ( LS k ) and the right amplitude of the audio source ( RS k Box 1102 can be followed by box 1104.

[0098] In box 1104, determine LS k and RS k The difference in amplitude between them D k Box 1104 can be followed by box 1106.

[0099] In box 1106, calculate D. k time derivative d(D k ) Box 1106 can be followed by box 1108.

[0100] In box 1108, determine LS k and RS k The average value is used to obtain the mid-channel spectrum ( MCSk Box 1108 can be followed by box 1110.

[0101] In box 1110, the sum of the mid-channel spectrograms of each of the separated audio sources (MCS1 + MCS2 + … + MCS) is used for separation. n ) to MCS k Normalization is performed to obtain the normalized value (R). k Box 1110 can be followed by box 1112.

[0102] In box 1112, d(D) K Divide by R k To obtain a confidence map, wherein different regions of the confidence map are associated with corresponding probability values ​​indicating the likelihood that the region corresponds to a spatial artifact. Box 1112 may be followed by box 1114.

[0103] In box 1114, the mixed weights are calculated by expanding and cropping the confidence plot. Box 1216 can follow box 1114.

[0104] In box 1116, combine MCS k Mixed weights and L t To obtain the modified left channel and combine the MCS k Mixed weights and R t To obtain the modified right channel.

[0105] In some embodiments, method 1100 further includes performing a modified summation of the left channels of two or more audio sources to obtain a left output channel, and performing a modified summation of the right channels of two or more audio sources to obtain a right output channel. In some embodiments, method 1100 further includes receiving a command to erase a specific audio source, wherein two or more audio sources exclude the specific audio source. In some embodiments, performing the summation includes applying a corresponding weight to each of the two or more audio sources.

[0106] In some embodiments, method 1100 further includes performing an inverse short-time Fourier transform (STFT) on the left output channel to obtain a left playback channel, and performing an inverse short-time Fourier transform on the right output channel to obtain a right playback channel, wherein the left playback channel and the right playback channel can be used to output audio via a speaker.

[0107] In addition to the description above, users can be provided with controls that allow them to choose whether and when the system, program, or feature described herein enables the collection of user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current location) and whether to send content or communications to the user from the server. Furthermore, certain data can be processed in one or more ways before being stored or used to remove personally identifiable information. For example, a user's identity can be processed to make it impossible to determine the user's personally identifiable information, or, if location information is available, the user's geographic location can be generalized (e.g., to the city, zip code, or state level) to make it impossible to determine the user's specific location. Therefore, users can control what information is collected from them, how that information is used, and what information is provided to them.

[0108] In the foregoing description, numerous specific details have been set forth for purposes of explanation in order to provide a thorough understanding of this specification. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In some instances, structures and apparatuses have been shown in block diagram form to avoid obscuring the description. For example, embodiments may be described above primarily with reference to user interfaces and specific hardware. However, embodiments may be applied to any type of computing device capable of receiving data and commands, as well as any peripheral devices providing services.

[0109] The references to "some embodiments" or "some examples" in this specification mean that a particular feature, structure, or characteristic described in connection with an embodiment or example may be included in at least one implementation of this description. The phrase "in some embodiments" appearing in various places throughout this specification does not necessarily refer to the same embodiment.

[0110] Some parts of the detailed description above are presented in terms of algorithms and symbolic representations of operations on data bits within computer memory. These algorithmic descriptions and representations are means by which those skilled in the art of data processing most effectively communicate the substance of their work to others skilled in the art. An algorithm here, and generally is, considered a self-consistent sequence of steps that produces a desired result. These steps are those that require physical manipulation of physical quantities. Although not essential, these quantities are typically in the form of electrical or magnetic data that can be stored, transmitted, combined, compared, and otherwise manipulated. It has been found that, primarily for reasons of general use, these data may sometimes be appropriately referred to as bits, values, elements, symbols, characters, items, numbers, etc.

[0111] However, it should be remembered that all these and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise specifically indicated, as will be apparent from the following discussion, it should be understood that throughout the description, the use of terms including “processing” or “calculating” or “operating” or “determining” or “displaying” refers to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the registers and memory of the computer system and transforms that data into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0112] Embodiments of this specification may also relate to a processor for performing one or more steps of the methods described above. This processor may be a dedicated processor that is selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including but not limited to: any type of disk (including optical disks), ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic cards or optical cards, flash memory (including USB flash drives with non-volatile memory), or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0113] This specification may take the form of some entirely hardware embodiments, some entirely software embodiments, or some embodiments that include both hardware and software elements. In some embodiments, this specification is implemented in software, including but not limited to firmware, resident software, microcode, etc.

[0114] Furthermore, this description may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in conjunction with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium may be any device capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0115] A data processing system suitable for storing or executing program code will include at least one processor directly or indirectly coupled to memory elements via a system bus. Memory elements may include local memory, mass storage, and cache memory used during the actual execution of the program code, the cache memory providing temporary storage for at least some of the program code to reduce the number of times code must be retrieved from mass storage during execution.

Claims

1. A method for modifying audio sources, including having a corresponding left (…) L t ) channel and right ( R t Multiple channels n A computer-implemented method for an audio source audio stream, wherein the audio source is separated from the original audio stream, wherein the L t The vocal tract and the R t The method, which represents the audio channel in a time-frequency manner, includes, for each audio source ( k ): Determine the left amplitude value of the audio source ( LS k ) and the right amplitude of the audio source ( RS k ); Determine the LS k and the RS k The difference in amplitude between them D k ); Calculate the D k time derivative d(D k ) ; Determine LS k and RS k The average value is used to obtain the mid-channel spectrum ( MCS k ); Based on the sum of the mid-channel spectrograms of each of the separated audio sources (MCS1 + MCS2 + ... + MCS) n ) to the MCS k Perform normalization to obtain the normalized value R. k ; d(D) K Divide by R k To obtain a confidence map, wherein different regions of the confidence map are associated with corresponding probability values ​​indicating the probability that the region corresponds to a spatial artifact; The mixed weights are calculated by scaling and cropping the confidence graph; and Combined MCS k The mixed weights and the L t To obtain the modified left channel, and to combine the MCS k The mixed weights and the R t To obtain the modified right channel.

2. The method of claim 1, further comprising performing a summation of the modified left channels of two or more audio sources in the audio sources to obtain a left output channel, and performing a summation of the modified right channels of the two or more audio sources in the audio sources to obtain a right output channel.

3. The method of claim 2, further comprising performing an inverse short-time Fourier transform (STFT) on the left output channel to obtain a left playback channel, and performing the STFT on the right output channel to obtain a right playback channel, wherein the left playback channel and the right playback channel are usable for outputting audio via a speaker.

4. The method of claim 3, further comprising receiving a command to erase a specific audio source, wherein two or more of the audio sources are excluded from the left playback channel and the right playback channel.

5. The method of claim 2, wherein, Performing the summation involves applying a corresponding weight to each of the two or more audio sources.

6. The method of claim 1, further comprising: Separate the original audio stream from the video; The original audio stream is provided as input to the source separation model; as well as The source separation model is used to output the audio stream, which includes the multiple audio sources.

7. The method of claim 1, further comprising, upon determining the LS k and the RS k Before: Receive the raw audio stream, the raw audio stream including the left ( L ) signal and right ( R )Signal; The short-time Fourier transform (STFT) is applied to the L signal and the R signal respectively to obtain the left channel L. st and the right channel R st ; Combining the L st and the R st ; Applying the source separation model to the combined L st and R st To obtain a corresponding mask for each of the plurality of audio sources; as well as Execute the corresponding mask and the L st and the R st Pointwise multiplication is performed to obtain the corresponding value for each audio source. L t and stated R t The aforementioned multiple audio sources.

8. The method of claim 7, wherein, Combining the L st and the R st include: Calculate the L st and the R st The average value; and Calculate the magnitude of the average value.

9. A non-transitory computer-readable medium for modifying a device comprising a corresponding left (or right) for each audio source. L t ) channel and right ( R t Multiple channels n The audio stream of the audio source, which is separate from the original audio stream, wherein the L t The vocal tract and the R t The audio channels are represented in a time-frequency manner, and instructions are stored on the non-transitory computer-readable medium, which, when executed by one or more computers, cause the one or more computers to perform operations, including, for each audio source ( k ): Determine the left amplitude value of the audio source ( LS k ) and the right amplitude of the audio source ( RS k ); Determine the LS k and the RS k The difference in amplitude between them D k ); Calculate the D k time derivative d(D k ) ; Determine LS k and RS k The average value is used to obtain the mid-channel spectrum ( MCS k ); Based on the sum of the mid-channel spectrograms of each of the separated audio sources (MCS1 + MCS2 + ... + MCS) n ) to the MCS k Perform normalization to obtain the normalized value R. k ; d(D) K Divide by R k To obtain a confidence map, wherein different regions of the confidence map are associated with corresponding probability values ​​indicating the probability that the region corresponds to a spatial artifact; The mixed weights are calculated by scaling and cropping the confidence graph; and Combined MCS k The mixed weights and the L t To obtain the modified left channel, and to combine the MCS k The mixed weights and the R t To obtain the modified right channel.

10. The non-transitory computer-readable medium of claim 9, wherein, The operation further includes performing a summation of the modified left channels of two or more audio sources in the audio sources to obtain a left output channel, and performing a summation of the modified right channels of the two or more audio sources in the audio sources to obtain a right output channel.

11. The non-transitory computer-readable medium of claim 9, wherein, The operation further includes performing an inverse short-time Fourier transform (STFT) on the left output channel to obtain a left playback channel, and performing the STFT on the right output channel to obtain a right playback channel, wherein the left playback channel and the right playback channel can be used to output audio via a speaker.

12. The non-transitory computer-readable medium of claim 11, wherein, The operation further includes receiving a command to erase a specific audio source, wherein two or more audio sources are excluded from the left playback channel and the right playback channel.

13. The non-transitory computer-readable medium of claim 10, wherein, Performing the summation involves applying a corresponding weight to each of the two or more audio sources.

14. The non-transitory computer-readable medium of claim 9, wherein the operation further comprises: Separate the original audio stream from the video; The original audio stream is provided as input to the source separation model; as well as The source separation model is used to output the audio stream, which includes the multiple audio sources.

15. The non-transitory computer-readable medium of claim 9, wherein, The operation further includes determining the LS k and the RS k Before: Receive the raw audio stream, the raw audio stream including the left ( L ) signal and right ( R )Signal; The short-time Fourier transform (STFT) is applied to the L signal and the R signal respectively to obtain the left channel L. st and the right channel R st ; Combining the L st and the R st ; Applying the source separation model to the combined L st and R st To obtain a corresponding mask for each of the plurality of audio sources; as well as Execute the corresponding mask and the L st and the R st Pointwise multiplication is performed to obtain the corresponding value for each audio source. L t and stated R t The aforementioned multiple audio sources.

16. A method for modifying audio sources, including having a corresponding left (…) for each audio source. L t ) channel and right ( R t Multiple channels n A computing device for an audio stream from an audio source, wherein the audio source is separate from the original audio stream, and wherein the L t The vocal tract and the R t The audio channels are represented in a time-frequency manner, and the computing device includes: processor; as well as A memory coupled to the processor stores instructions that, when executed by the processor, cause the processor to perform operations, including, for each audio source ( k ): Determine the left amplitude value of the audio source ( LS k ) and the right amplitude of the audio source ( RS k ); Determine the LS k and the RS k The difference in amplitude between them D k ); Calculate the D k time derivative d(D k ) ; Determine LS k and RS k The average value is used to obtain the mid-channel spectrum ( MCS k ); Based on the sum of the mid-channel spectrograms of each of the separated audio sources (MCS1 + MCS2 + ... + MCS) n ) to the MCS k Perform normalization to obtain the normalized value R. k ; d(D) K Divide by R k To obtain a confidence map, wherein different regions of the confidence map are associated with corresponding probability values ​​indicating the probability that the region corresponds to a spatial artifact; The mixed weights are calculated by scaling and cropping the confidence graph; and Combined MCS k The mixed weights and the L t To obtain the modified left channel, and to combine the MCS k The mixed weights and the R t To obtain the modified right channel.

17. The computing device of claim 16, wherein, The operation further includes performing a summation of the modified left channels of two or more audio sources in the audio sources to obtain a left output channel, and performing a summation of the modified right channels of the two or more audio sources in the audio sources to obtain a right output channel.

18. The computing device of claim 16, wherein, The operation further includes performing an inverse short-time Fourier transform (STFT) on the left output channel to obtain a left playback channel, and performing the STFT on the right output channel to obtain a right playback channel, wherein the left playback channel and the right playback channel can be used to output audio via a speaker.

19. The computing device of claim 18, wherein, The operation further includes receiving a command to erase a specific audio source, wherein two or more audio sources are excluded from the left playback channel and the right playback channel.

20. The computing device of claim 17, wherein, Performing the summation involves applying a corresponding weight to each of the two or more audio sources.

Images