A neural network algorithm implemented method and system for separating vocals and instrumental sounds
By using neural network algorithms and bidirectional long short-term memory neural networks, the problem of separating human voices from musical sounds in a closed reverberation environment was solved, achieving efficient separation in a closed acoustic environment and reducing training costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN VIMAI TECH CO LTD
- Filing Date
- 2026-04-30
- Publication Date
- 2026-07-31
AI Technical Summary
In a closed reverberant environment, traditional sound source separation methods struggle to effectively distinguish between human voices and musical sounds, while deep learning methods require a large amount of training data and complex array configurations, resulting in high costs for data construction and model training.
A neural network algorithm is used to perform time-frequency analysis on the mixed microphone signals through a bidirectional long short-term memory neural network. Short-time Fourier transform and clustering techniques are used to separate human voices and musical sounds, reducing the computational burden caused by the expansion of the training sample size.
This method achieves effective separation of human voices and musical sounds under overlapping sound source conditions, is applicable to closed acoustic environments, reduces training costs, and improves the model's separation capability.
Smart Images
Figure CN122493877A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of sound source separation technology, and in particular to a method and system for separating human voice and musical sound using a neural network algorithm. Background Technology
[0002] In applications such as voice communication, audio enhancement, remote recognition, human-computer interaction, voice entertainment, and sound source localization, the signals collected by microphones are often not from a single sound source, but rather a mixture of signals formed by the superposition of multiple sound sources after propagation, reflection, and reverberation within an indoor environment. Traditional processing methods based on beamforming, time difference of arrival, cross-correlation, or spatial spectrum estimation typically rely on idealized free-field propagation assumptions, which can easily degrade in performance under enclosed spaces, reverberation, and noise conditions. While existing deep learning methods can learn complex acoustic features through training, when processing multi-source signals with simultaneous emission and high spectral overlap, they often require extensive training data covering source locations, room parameters, reverberation conditions, and energy relationships, resulting in high costs for data construction and model training. Summary of the Invention
[0003] In view of the above technical problems, the present invention provides a method and system for separating human voice and music sound using a neural network algorithm. It aims to solve the problem that traditional sound source separation or localization methods are difficult to effectively distinguish different sound sources when human voice and music sound overlap in the time-frequency domain in a mixed microphone signal in a closed reverberation environment, and that deep learning multi-sound source processing usually relies on large-scale training data and complex array configuration.
[0004] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.
[0005] According to one aspect of the present invention, a method for separating human voice and musical vocals using a neural network algorithm is proposed, the method comprising: Acquire a mixed microphone signal, the mixed microphone signal comprising a human voice propagation component formed by the propagation of a human voice source through the corresponding room impulse response, a curved sound propagation component formed by the propagation of a curved sound source through the corresponding room impulse response, and a noise component; Perform a short-time Fourier transform on each of the hybrid microphone signals to obtain the hybrid amplitude spectrum and hybrid phase spectrum corresponding to the hybrid microphone signal, and organize the hybrid amplitude spectrum into neural network input features according to time frames and frequency units; The neural network input features are input into a trained bidirectional long short-term memory neural network separation model. The bidirectional long short-term memory neural network separation model performs temporal modeling on the neural network input features along the forward and reverse time directions, and outputs the embedding representation corresponding to each time frequency unit. Clustering is performed on the embedded representation to obtain estimated binary masks corresponding to human voice and musical voice classes, respectively. Based on the estimated binary masks, the mixed amplitude spectrum is divided into human voice amplitude spectrum and musical voice amplitude spectrum. The human voice amplitude spectrum and the mixed phase spectrum are combined and then subjected to inverse short-time Fourier transform to reconstruct the human voice time-domain signal. The music amplitude spectrum and the mixed phase spectrum are combined and then subjected to inverse short-time Fourier transform to reconstruct the music time-domain signal, thereby separating the human voice and music in the mixed microphone signal.
[0006] Furthermore, before performing a short-time Fourier transform on each of the hybrid microphone signals, the hybrid microphone signals are subjected to frame segmentation, windowing, and adjacent frame overlap processing, so that the hybrid microphone signals are represented as multiple time-frequency units; the windowing uses a smooth window function to reduce the influence of frame boundaries on the spectral representation; amplitude information and phase information are extracted from the time-frequency units respectively, the amplitude information is used as the input feature of the neural network, the phase information is retained as the phase reference, and the neural network input features are divided into feature blocks of uniform size for batch training and inference of the bidirectional long short-term memory neural network separation model.
[0007] Furthermore, the bidirectional long short-term memory neural network separation model includes a sequentially connected input layer, several bidirectional long short-term memory layers, a random deactivation regularization layer, a fully connected output layer, a nonlinear activation layer, and a norm regularization layer. The bidirectional long short-term memory layer includes long short-term memory units that process the neural network input features forward in time and long short-term memory units that process the neural network input features backward in time, to simultaneously learn the temporal correlation of human voices and the temporal correlation of musical notes between consecutive time frames in the mixed microphone signal. The random deactivation regularization layer is used to suppress overfitting of the model to training samples. The fully connected output layer is used to map the temporal representation output by the bidirectional long short-term memory layer to the embedded representation of each time-frequency unit. The nonlinear activation layer is used to constrain the embedded representation to a value form that facilitates masking. The norm regularization layer is used to improve the stability of the embedded representation during the clustering stage.
[0008] Furthermore, the training objective of the bidirectional long short-term memory neural network separation model is ideal binary masking. Ideal binary masking is determined by comparing the relative dominance of the spectral amplitude of the human voice source and the spectral amplitude of the musical voice source within the same time-frequency unit in the training samples. When the spectral amplitude of the human voice source is dominant, the corresponding time-frequency unit is labeled as the human voice class; when the spectral amplitude of the musical voice source is dominant, the corresponding time-frequency unit is labeled as the musical voice class. During training, the loss is constructed based on the difference between the estimated affinity formed by the embedding representation output by the network and the target affinity formed by the ideal binary masking. An adaptive moment estimation optimization algorithm is used to update the model parameters through backpropagation, so that time-frequency units belonging to the same sound source category are closer to each other in the embedding space, and time-frequency units belonging to different sound source categories are farther apart in the embedding space.
[0009] Furthermore, the training samples are obtained by acquiring human voice audio and music audio from the training audio set, generating room impulse responses from the sound source position to each microphone position according to preset sound source direction and acoustic space parameters, convolving the human voice audio and the music audio with the corresponding room impulse responses to obtain single-source propagation signals received by each microphone, and superimposing the single-source propagation signals according to different energy relationships in the same microphone channel to obtain a mixed training signal; the room impulse responses are generated by the image sound source method based on the size of the closed acoustic space, the microphone array position, the sound source position and the reverberation time; the amplitude spectrum is extracted from the mixed training signal as training input, and the ideal binary mask is generated from the human voice spectrum and music spectrum that retain the category identification, so that the bidirectional long short-term memory neural network separation model learns the ability to separate overlapping human voice and music under room reverberation conditions.
[0010] Furthermore, clustering the embedded representation includes treating the embedded representation as sample points in the embedding space, using the mean clustering algorithm to divide the sample points into human voice clusters and musical voice clusters, and rearranging them according to the category labels obtained from the clustering into the estimated binary mask corresponding to the mixed amplitude spectrum; the masking portion of the estimated binary mask corresponding to the human voice cluster is used to retain the time frequency units belonging to the human voice in the mixed amplitude spectrum, and the masking portion of the estimated binary mask corresponding to the musical voice cluster is used to retain the time frequency units belonging to the musical voice in the mixed amplitude spectrum.
[0011] Furthermore, when dividing the mixed amplitude spectrum into a vocal amplitude spectrum and a musical amplitude spectrum based on the estimated binary mask, the vocal amplitude spectrum is obtained by combining the retained portion corresponding to the estimated binary mask with the mixed amplitude spectrum unit by unit, and the musical amplitude spectrum is obtained by combining the complementary portion corresponding to the estimated binary mask with the mixed amplitude spectrum unit by unit. The vocal amplitude spectrum and the musical amplitude spectrum are respectively combined with the mixed phase spectrum to form a complex spectrum, and then synthesized by inverse short-time Fourier transform and inter-frame overlap to obtain the vocal time-domain signal and the musical time-domain signal corresponding to each microphone channel.
[0012] Furthermore, after obtaining the human voice time-domain signal and the music time-domain signal from each microphone channel, the human voice multi-channel signal composed of the reconstructed signals from each human voice source and the music multi-channel signal composed of the reconstructed signals from each music source are respectively input into the pre-trained single-source localization model. The single-source localization model extracts sound intensity features and generalized cross-correlation phase transformation features from the human voice multi-channel signal and the music multi-channel signal, and represents the sound source direction as a discrete direction category through a convolutional neural network, thereby outputting the human voice source direction and the music source direction while separating the human voice and the music.
[0013] According to another aspect of the present invention, a vocal and instrumental voice separation system implemented using a neural network algorithm is provided, comprising: The acquisition module is used to acquire a mixed microphone signal, which includes a human voice propagation component formed by the propagation of a human voice source through the corresponding room impulse response, a curved sound propagation component formed by the propagation of a curved sound source through the corresponding room impulse response, and a noise component. The feature transformation module is used to perform short-time Fourier transform on each of the hybrid microphone signals to obtain the hybrid amplitude spectrum and hybrid phase spectrum corresponding to the hybrid microphone signal, and to organize the hybrid amplitude spectrum into neural network input features according to time frames and frequency units; The feature transformation module is used to input the neural network input features into a trained bidirectional long short-term memory neural network separation model, and to perform temporal modeling on the neural network input features along the forward and reverse time directions through the bidirectional long short-term memory neural network separation model, and output the embedding representation corresponding to each time frequency unit; The clustering module is used to cluster the embedded representation to obtain estimated binary masks corresponding to human voice and musical voice classes, respectively, and to divide the mixed amplitude spectrum into human voice amplitude spectrum and musical voice amplitude spectrum based on the estimated binary masks. The separation module is used to combine the human voice amplitude spectrum with the mixed phase spectrum and then perform an inverse short-time Fourier transform to reconstruct the human voice time-domain signal, and to combine the music voice amplitude spectrum with the mixed phase spectrum and then perform an inverse short-time Fourier transform to reconstruct the music voice time-domain signal, thereby separating the human voice and music voice in the mixed microphone signal.
[0014] The technical solution of the present invention has the following beneficial effects: This invention performs time-frequency analysis on mixed microphone signals, using the amplitude spectrum as input to a neural network and employing a bidirectional long short-term memory network to model the correlation between consecutive time frames. This improves the model's ability to represent the temporal differences between human voices and musical sounds. By embedding representations to cluster and form binary masks corresponding to different sound source categories, the mixed amplitude spectrum can be divided into human voice amplitude spectrum and musical sound amplitude spectrum. The corresponding time-domain signal is then reconstructed by combining the preserved mixed phase spectrum, thereby achieving separation of human voices and musical sounds under overlapping sound source conditions. This invention does not rely on directly enumerating all possible combinations of multiple sound sources for classification, reducing the computational burden caused by the expansion of training sample size in multi-source overlapping scenarios, and is applicable to enclosed acoustic environments where room impulse response is a factor. Attached Figure Description
[0015] Figure 1 This is a flowchart of a method for separating human voice and musical voice implemented using a neural network algorithm in an embodiment of this specification; Figure 2 This is a structural block diagram of a vocal and instrumental sound separation system implemented using a neural network algorithm in an embodiment of this specification. Detailed Implementation
[0016] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the invention more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention may be practiced with one or more of these specific details omitted, or other methods, components, systems, steps, etc., may be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the invention.
[0017] Furthermore, the accompanying drawings are merely illustrative of the invention. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.
[0018] This invention provides a method for separating human voice and musical vocals using a neural network algorithm. (Refer to...) Figure 1 The diagram shown is a flowchart illustrating a method for separating human voice and musical vocals using a neural network algorithm according to an embodiment of the present invention. This method is implemented in an Android all-in-one advertising screen, particularly a battery-powered Android all-in-one advertising screen terminal. Specifically, the method may include the following steps S101~S105: In step S101, a mixed microphone signal is acquired, which includes a human voice propagation component formed by the propagation of a human voice source through the corresponding room impulse response, a chordal sound propagation component formed by the propagation of a chordal sound source through the corresponding room impulse response, and a noise component.
[0019] Acquire the mixed microphone signal collected by the microphone array. This mixed microphone signal is the superposition signal of human voice and acoustic sound sources emitted simultaneously in a closed acoustic space, reaching their respective microphones. During propagation, the human voice and acoustic sound sources are affected by the room structure, sound source location, microphone location, and reflection path, and after being processed by the corresponding room impulse response, they form the human voice propagation component and the acoustic sound propagation component reaching the microphone. There are multiple microphones, and the signal received by each microphone may also contain background noise and microphone noise. Therefore, the i-th mixed microphone signal can be represented as: ; in, This represents the mixed microphone signal received by the i-th microphone. Represents the human voice source signal. Indicates the sound source signal of the melody. This represents the room impulse response between the human sound source and the i-th microphone. This represents the room impulse response between the sound source and the i-th microphone. Let represent the noise component in the i-th microphone channel, and * denote the convolution operation. Since the acquired signal is a digital signal, the above propagation superposition relationship can also be written as: ; in, and Here, T is the discrete-time index, and T is the effective length of the room impulse response. Therefore, the mixed microphone signal is not simply the sum of the amplitudes of the human voice and the music source signals, but rather the result of convolving the two types of sound sources through their respective spatial propagation paths, followed by the superposition of noise. This signal serves as the initial input for subsequent human and music sound separation processing, enabling subsequent processing to address the actual microphone reception signal affected by reverberation propagation.
[0020] In step S102, a short-time Fourier transform is performed on each of the hybrid microphone signals to obtain the hybrid amplitude spectrum and hybrid phase spectrum corresponding to the hybrid microphone signal. The hybrid amplitude spectrum is then organized into neural network input features according to time frames and frequency units.
[0021] Before performing a short-time Fourier transform on each of the mixed microphone signals, the mixed microphone signals are subjected to framing, windowing, and adjacent frame overlap processing, so that the mixed microphone signals are represented as multiple time-frequency units. The windowing uses a smooth window function to reduce the impact of frame boundaries on the spectral representation. Amplitude information and phase information are extracted from the time-frequency units respectively. The amplitude information is used as the input feature of the neural network, the phase information is retained as the phase reference, and the neural network input features are divided into feature blocks of uniform size for batch training and inference of the bidirectional long short-term memory neural network separation model.
[0022] In this process, a short-time Fourier transform (SFT) is performed on each mixed microphone signal to convert the time-domain form of the mixed microphone signal into a time-frequency domain representation. The SFT uses a fixed-length time window to perform local analysis of the audio signal, allowing each short-time segment to be approximated as a stationary signal within that window, thus obtaining the corresponding spectral representation. To reduce the impact of spectral abrupt changes caused by frame boundaries, a smoothing window function such as a Hanning window can be applied to each frame after framing, and a certain overlap between adjacent time frames is maintained to improve the continuity of the time-frequency representation.
[0023] After short-time Fourier transform, each mixed microphone signal is represented as multiple time-frequency units, each containing amplitude and phase information. Amplitude information characterizes the energy distribution of the mixed microphone signal within the corresponding time frame and frequency unit, serving as input features to the bidirectional long short-term memory neural network separation model. Phase information is not input into the model for learning but is retained as a phase reference for subsequent time-domain signal reconstruction. This approach allows the neural network to focus on learning the differences in amplitude spectra between the human voice propagation component and the melody propagation component, while avoiding increased model complexity caused by directly estimating the phase during the separation stage.
[0024] In one alternative implementation, the short-time Fourier transform employs a 256-point Hanning window, corresponding to a window length of 16 milliseconds, resulting in 129 frequency units. A 25% overlap can be set between adjacent time segments to expand the time-frequency samples available for model processing and maintain a continuous transition between adjacent frames. The resulting mixed amplitude spectrum, arranged according to time frames and frequency units, can be divided into feature blocks of uniform size; for example, each feature block includes 100 time frames and 129 frequency units, giving the neural network input features a unified dimension. This feature block can serve as the basic input unit for batch training and inference, thereby ensuring that the bidirectional long short-term memory neural network separation model can perform temporal modeling of the mixed amplitude spectrum under a consistent data structure.
[0025] In step S103, the neural network input features are input into a trained bidirectional long short-term memory neural network separation model. The bidirectional long short-term memory neural network separation model performs time-series modeling on the neural network input features along the forward and reverse time directions, and outputs the embedding representation corresponding to each time frequency unit.
[0026] The bidirectional long short-term memory (LSTM) neural network separation model includes a sequentially connected input layer, several bidirectional LSTM layers, a random deactivation regularization layer, a fully connected output layer, a nonlinear activation layer, and a norm regularization layer. The bidirectional LSTM layers include LSTM units that process the neural network input features forward in time and LSTM units that process the neural network input features backward in time, to simultaneously learn the temporal correlation of human voices and the temporal correlation of musical notes between consecutive time frames in the mixed microphone signal. The random deactivation regularization layer is used to suppress overfitting of the model to training samples. The fully connected output layer is used to map the temporal representation output by the bidirectional LSTM layers to the embedded representation of each time-frequency unit. The nonlinear activation layer is used to constrain the embedded representation to a value form that facilitates masking. The norm regularization layer is used to improve the stability of the embedded representation during the clustering stage.
[0027] The training objective of the bidirectional long short-term memory neural network separation model is ideal binary masking. Ideal binary masking is determined by comparing the relative dominance of the spectral amplitude of the human voice source and the spectral amplitude of the musical voice source within the same time-frequency unit in the training samples. When the spectral amplitude of the human voice source is dominant, the corresponding time-frequency unit is labeled as the human voice class; when the spectral amplitude of the musical voice source is dominant, the corresponding time-frequency unit is labeled as the musical voice class. During training, the loss is constructed based on the difference between the estimated affinity formed by the embedding representation output by the network and the target affinity formed by the ideal binary masking. An adaptive moment estimation optimization algorithm is used to update the model parameters through backpropagation, so that time-frequency units belonging to the same sound source category are closer to each other in the embedding space, and time-frequency units belonging to different sound source categories are farther apart in the embedding space.
[0028] The training samples are obtained by acquiring human voice and music audio from the training audio set, generating room impulse responses from the sound source location to each microphone location according to preset sound source direction and acoustic space parameters, and convolving the human voice and music audio with the corresponding room impulse responses to obtain single-source propagation signals received by each microphone. The single-source propagation signals are then superimposed into a mixed training signal within the same microphone channel according to different energy relationships. The room impulse responses are generated using the image sound source method based on the size of the enclosed acoustic space, the microphone array location, the sound source location, and the reverberation time. The amplitude spectrum is extracted from the mixed training signal as training input, and the ideal binary mask is generated from the human voice and music spectra that retain the category labels, thereby enabling the bidirectional long short-term memory neural network separation model to learn the ability to separate overlapping human voices and music under room reverberation conditions.
[0029] Specifically, in step S103, the bidirectional long short-term memory neural network belongs to the recurrent neural network structure, which is suitable for modeling time-varying sequence data and can alleviate the gradient vanishing problem that ordinary recurrent neural networks are prone to in long sequence training. For mixed microphone signals where human voice and musical sound overlap in the time-frequency domain, the bidirectional long short-term memory neural network separates the model to process the input features simultaneously along the forward and backward time directions, so that the model not only utilizes the spectral change relationship before the current time frame, but also the spectral change relationship after the current time frame, thereby obtaining more complete temporal context information.
[0030] In one alternative implementation, the input features are sized to 100 time frames and 129 frequency units per batch. Two bidirectional long short-term memory (LSSM) layers are connected after the input layer. Each bidirectional LSM layer includes LSM units processed in the forward time direction and LSM units processed in the backward time direction, with 300 neurons in each direction. A random deactivation regularization layer is applied after each bidirectional LSM layer to reduce the model's dependence on training samples and suppress overfitting during training. The temporal representation output from the bidirectional LSM layers is fed into a fully connected output layer, which maps the temporal representation to the embedding representation of each time-frequency unit. The fully connected output layer can have an output dimension of 129 x 3, so that each time-frequency unit corresponds to a three-dimensional embedding vector. The nonlinear activation layer can use the sigmoid function to make the output value range between zero and one, so as to form a mask later. L2 norm regularization is applied after the nonlinear activation layer to make the output embedding have a more stable scale relationship during clustering. The model output can be represented as a matrix. 12,900 corresponds to the number of combinations of 100 time frames and 129 frequency units, and 3 represents the embedding dimension of each time frequency unit.
[0031] The bidirectional long short-term memory neural network separation model uses ideal binary masking as the training objective. Ideal binary masking is determined by comparing the relative dominance of the amplitudes of human and curved sound source spectra within the same time frequency unit, and is defined as follows: ; in, This represents the spectral representation of a human sound source at time frame t and frequency unit f. This represents the spectral representation of a sound source at the same time frequency unit. When the spectral amplitude of a human voice source is dominant relative to that of a sound source, the corresponding time frequency unit is labeled as the human voice class; when the spectral amplitude of a sound source is dominant, the corresponding time frequency unit is labeled as the sound source class. This ideal binary mask is used to construct target class relationships, enabling the model to learn to map time frequency units belonging to the same sound source class to nearby positions in the embedding space, and to map time frequency units belonging to different sound source classes to distant positions.
[0032] During training, the network output embeddings form the estimated affinity, the ideal binary mask forms the target affinity, and the loss is determined by the difference between the two, which can be expressed as: ; Where V represents the embedding representation matrix output by the network, and Y represents the target matrix formed by the ideal binary mask. This represents the estimation of the affinity matrix. Represents the target affinity matrix. Let represent the Frobenius norm. For ease of calculation, this loss can also be written as: ; By minimizing the aforementioned loss, the embedded representation of the network output gradually satisfies the sound source category clustering relationship. The model parameters can be optimized using an adaptive moment estimation algorithm and updated via backpropagation. In one optional implementation, the initial learning rate of the adaptive moment estimation algorithm is 0.01, the number of training epochs is 50, and the batch size is 64 samples.
[0033] Training samples can be obtained by constructing a mixture of human voice and instrumental sound signals within a closed acoustic space. Human voice and instrumental sound audio can be selected from the training audio set as two independent sound source signals. Room impulse responses from the sound sources to each microphone are generated based on preset sound source directions, positions, microphone positions, dimensions of the closed acoustic space, and reverberation time. The room impulse responses can be generated using an image-based sound source method to simulate the propagation process of sound sources within a closed space after reflection and reverberation before reaching the microphones. The human voice and instrumental sound audio are convolved with their corresponding room impulse responses to obtain single-source human voice and instrumental sound propagation signals within each microphone channel. These signals are then superimposed within the same microphone channel to form a mixed training signal. The amplitude spectrum is extracted from the mixed training signal as training input. An ideal binary mask is generated from the human voice and instrumental sound spectra with category labels, enabling the bidirectional long short-term memory neural network separation model to learn separable embedding representations of overlapping human and instrumental sounds in the time-frequency domain under closed reverberation conditions.
[0034] In step S104, the embedded representation is clustered to obtain estimated binary masks corresponding to human voice and musical voice classes, respectively, and the mixed amplitude spectrum is divided into human voice amplitude spectrum and musical voice amplitude spectrum based on the estimated binary masks.
[0035] Clustering the embedded representation includes treating the embedded representation as sample points in the embedding space, using the mean clustering algorithm to divide the sample points into human voice clusters and musical voice clusters, and rearranging them according to the category labels obtained from the clustering into the estimated binary mask corresponding to the mixed amplitude spectrum; the masking part of the estimated binary mask corresponding to the human voice cluster is used to retain the time frequency units belonging to the human voice in the mixed amplitude spectrum, and the masking part of the estimated binary mask corresponding to the musical voice cluster is used to retain the time frequency units belonging to the musical voice in the mixed amplitude spectrum.
[0036] When the mixed amplitude spectrum is divided into a vocal amplitude spectrum and a musical amplitude spectrum based on the estimated binary mask, the vocal amplitude spectrum is obtained by combining the retained portion corresponding to the estimated binary mask with the mixed amplitude spectrum unit by unit, and the musical amplitude spectrum is obtained by combining the complementary portion corresponding to the estimated binary mask with the mixed amplitude spectrum unit by unit. The vocal amplitude spectrum and the musical amplitude spectrum are respectively combined with the mixed phase spectrum to form a complex spectrum, and then synthesized by inverse short-time Fourier transform and inter-frame overlap to obtain the vocal time-domain signal and the musical time-domain signal corresponding to each microphone channel.
[0037] Each sample point corresponds to a time-frequency unit in the mixed amplitude spectrum, and the position of the sample point is determined by the embedding vector learned by the model. Since the training process has caused time-frequency units of the same sound source category to tend to be closer in the embedding space, and time-frequency units of different sound source categories to tend to be farther apart, the K-means clustering algorithm can be used to divide all sample points into two categories, corresponding to the human voice cluster and the musical voice cluster, respectively.
[0038] After clustering, the category labels of each sample point are rearranged according to the original time frame and frequency unit positions to form an estimated binary mask with the same size as the mixed amplitude spectrum. In the estimated binary mask, positions belonging to the human voice cluster are used to preserve the time-frequency units dominated by human voices in the mixed amplitude spectrum, and positions belonging to the musical voice cluster are used to preserve the time-frequency units dominated by musical voices in the mixed amplitude spectrum. Thus, the neural network does not directly output the human voice amplitude spectrum and the musical voice amplitude spectrum, but instead outputs clusterable embedding representations, which are then used by the clustering labels to form a mask for spectrum partitioning.
[0039] Let X be the spectrum of the mixed microphone signal, and IBM be the estimated binary mask. Then the amplitude spectrum of the vocal signal and the amplitude spectrum of the instrumental signal can be obtained by the following formulas: ; ; in, Represents the amplitude spectrum of human voice. Represents the amplitude spectrum of the musical notes. This represents unit-by-unit multiplication. Time-frequency units with an estimated binary mask value of one are assigned to the vocal amplitude spectrum, while time-frequency units with an estimated binary mask value of zero are assigned to the chord amplitude spectrum through complementary masking. The mixed spectrum can be represented as: ; This relationship indicates that the mixed spectrum is formed by the superposition of the spectral components of the two sound sources. After dividing the mixed amplitude spectrum into units based on the estimated binary mask, amplitude spectrum representations corresponding to human voice sources and chord sources can be obtained respectively. Each microphone channel can be processed independently in the same way, so that the spectral components belonging to human voice sources and chord sources in each channel are extracted separately.
[0040] After obtaining the vocal amplitude spectrum and the instrumental amplitude spectrum, they are combined with the mixed phase spectrum retained in step S102 to form the corresponding vocal complex spectrum and instrumental complex spectrum. Since the phase information comes from the same mixed microphone signal, the reconstruction process maintains the same phase reference as the original microphone channel. Subsequently, inverse short-time Fourier transforms are performed on the vocal complex spectrum and the instrumental complex spectrum, and adjacent frames are overlapped and synthesized to obtain the vocal time-domain signal and the instrumental time-domain signal corresponding to that microphone channel. After performing the above clustering, masking partitioning, and time-domain reconstruction processing on each mixed microphone signal, the vocal reconstruction signal and instrumental reconstruction signal under multiple microphone channels can be obtained.
[0041] In step S105, the human voice amplitude spectrum and the mixed phase spectrum are combined and then subjected to inverse short-time Fourier transform to reconstruct the human voice time-domain signal. The music amplitude spectrum and the mixed phase spectrum are combined and then subjected to inverse short-time Fourier transform to reconstruct the music time-domain signal, thereby separating the human voice and music in the mixed microphone signal.
[0042] Specifically, the obtained human voice amplitude spectrum is combined with the mixed phase spectrum of the same mixed microphone signal to form a complex spectral representation of the corresponding human voice source. An inverse short-time Fourier transform is then performed on this complex spectrum to obtain the human voice time-domain signal in that microphone channel. Similarly, the instrumental sound amplitude spectrum is combined with the mixed phase spectrum of the same channel to form a complex spectral representation of the corresponding instrumental sound source. An inverse short-time Fourier transform is then performed on this complex spectrum to obtain the instrumental sound time-domain signal in that microphone channel.
[0043] The mixed phase spectrum is used as a phase reference during reconstruction, enabling the vocal amplitude spectrum and the instrumental amplitude spectrum to be recovered from the time-frequency domain to the time domain. Since the vocal amplitude spectrum and the instrumental amplitude spectrum are obtained from the mixed amplitude spectrum by estimating binary masks, respectively, they retain the time-frequency units dominated by the vocal source and the instrumental source, respectively. Therefore, the time-domain signals after the inverse short-time Fourier transform correspond to the vocal component and the instrumental component in the mixed microphone signal, respectively.
[0044] During the inverse short-time Fourier transform (ISFT), each time frame is superimposed and synthesized according to the framing, windowing, and adjacent frame overlap methods used in the IFT, ensuring the continuity of the reconstructed time-domain signal at frame boundaries. After performing the same amplitude and phase spectrum combination and inverse transform processing on each mixed microphone signal, the corresponding vocal and instrumental time-domain signals for each microphone channel are obtained, thus completing the separation of vocal and instrumental sounds in the mixed microphone signal.
[0045] In one embodiment, after obtaining the human voice time-domain signal and the music time-domain signal of each microphone channel, the human voice multi-channel signal composed of each reconstructed signal belonging to the human voice source and the music multi-channel signal composed of each reconstructed signal belonging to the music source are respectively input into a pre-trained single-source localization model; the single-source localization model extracts sound intensity features and generalized cross-correlation phase transformation features from the human voice multi-channel signal and the music multi-channel signal, and represents the sound source direction as a discrete direction category through a convolutional neural network, thereby outputting the human voice source direction and the music source direction while separating the human voice and the music.
[0046] In this process, after obtaining the human voice time-domain signal and the music time-domain signal from each microphone channel, the multi-channel reconstructed signals belonging to the human voice source are grouped into a human voice multi-channel signal according to the microphone channel order, and the multi-channel reconstructed signals belonging to the music sound source are grouped into a music sound multi-channel signal according to the same channel order. Since the human voice multi-channel signal and the music sound multi-channel signal already correspond to a single sound source, they can be input into a pre-trained single-source localization model for sound source direction estimation, thus transforming the multi-source localization problem into two single-source localization problems.
[0047] A single-source localization model can be implemented using a convolutional neural network, with source direction estimation structured as a classification task of discrete direction categories. For each set of multi-channel signals, the single-source localization model extracts sound intensity features and generalized cross-correlation phase transform features. Sound intensity features characterize the spatial energy propagation relationship in the signal received by the microphone array, while generalized cross-correlation phase transform features characterize the time delay correlation between different microphone channels. Both types of features serve as input to the convolutional neural network, enabling the model to determine the direction of sound source arrival using spatial information from the multi-channel signals.
[0048] In one alternative embodiment, the microphone array is a planar orthogonal array comprising four omnidirectional microphones, two of which are arranged along the x-axis and the other two along the y-axis, with the array center serving as a reference point for the sound source direction angle. The sound source direction can be expressed as an angle relative to the positive x-axis. The directional range is: ; A pre-trained single-source localization model can divide this angular range into several discrete directional categories. For example, with a spatial resolution of ten degrees, there are thirty-six candidate directional categories, which are evenly distributed within the aforementioned angular range. The convolutional neural network outputs the prediction results corresponding to each directional category; the predicted category corresponding to the multi-channel human voice signal is taken as the direction of the human voice source, and the predicted category corresponding to the multi-channel musical sound signal is taken as the direction of the musical sound source.
[0049] Through the above processing, the separated vocal and musical time-domain signals can not only be used as their respective audio outputs, but also further used for sound source direction estimation. Since the input to the single-source localization model is the multi-channel signal corresponding to a single sound source after separation, the interference on direction judgment when the two sound sources completely overlap in the time and frequency domains can be reduced, thus outputting the vocal and musical source directions while completing the separation of vocal and musical sounds.
[0050] Based on the same line of thought, such as Figure 2 As shown, a vocal and instrumental voice separation system implemented using a neural network algorithm is provided, comprising: Acquisition module 201 is used to acquire a mixed microphone signal, the mixed microphone signal including a human voice propagation component formed by the propagation of a human voice source through the corresponding room impulse response, a curved sound propagation component formed by the propagation of a curved sound source through the corresponding room impulse response, and a noise component; The feature transformation module 202 is used to perform a short-time Fourier transform on each of the hybrid microphone signals to obtain the hybrid amplitude spectrum and hybrid phase spectrum corresponding to the hybrid microphone signal, and to organize the hybrid amplitude spectrum into neural network input features according to time frames and frequency units. The feature conversion module 203 is used to input the neural network input features into a trained bidirectional long short-term memory neural network separation model, and to perform temporal modeling on the neural network input features along the forward and reverse time directions through the bidirectional long short-term memory neural network separation model, and output the embedding representation corresponding to each time frequency unit; Clustering module 204 is used to cluster the embedded representation to obtain estimated binary masks corresponding to human voice and musical voice classes respectively, and to divide the mixed amplitude spectrum into human voice amplitude spectrum and musical voice amplitude spectrum based on the estimated binary masks; The separation module 205 is used to combine the human voice amplitude spectrum with the mixed phase spectrum and then perform an inverse short-time Fourier transform to reconstruct the human voice time-domain signal, and to combine the music amplitude spectrum with the mixed phase spectrum and then perform an inverse short-time Fourier transform to reconstruct the music time-domain signal, thereby separating the human voice and music in the mixed microphone signal.
[0051] This system performs time-frequency analysis on mixed microphone signals, using the amplitude spectrum as input to a neural network. It utilizes a bidirectional long short-term memory network to model the correlation between consecutive time frames, improving the model's ability to represent the temporal differences between human voices and musical sounds. By embedding representations to cluster and form binary masks corresponding to different sound source categories, the mixed amplitude spectrum can be divided into human voice and musical sound amplitude spectra. Combined with the preserved mixed phase spectrum, the corresponding time-domain signal is reconstructed, thus achieving separation of human voices and musical sounds under overlapping sound source conditions. This invention does not rely on directly enumerating all possible multi-source combinations for classification, reducing the computational burden caused by the expansion of training sample size in multi-source overlapping scenarios, and is applicable to enclosed acoustic environments where room impulse response is a factor.
[0052] It should be noted that although several modules or units of the device for performing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0053] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.
[0054] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A method for separating human voice and musical vocals using a neural network algorithm, characterized in that, The method includes: Acquire a mixed microphone signal, the mixed microphone signal comprising a human voice propagation component formed by the propagation of a human voice source through the corresponding room impulse response, a curved sound propagation component formed by the propagation of a curved sound source through the corresponding room impulse response, and a noise component; Perform a short-time Fourier transform on each of the hybrid microphone signals to obtain the hybrid amplitude spectrum and hybrid phase spectrum corresponding to the hybrid microphone signal, and organize the hybrid amplitude spectrum into neural network input features according to time frames and frequency units; The neural network input features are input into a trained bidirectional long short-term memory neural network separation model. The bidirectional long short-term memory neural network separation model performs temporal modeling on the neural network input features along the forward and reverse time directions, and outputs the embedding representation corresponding to each time frequency unit. Clustering is performed on the embedded representation to obtain estimated binary masks corresponding to human voice and musical voice classes, respectively. Based on the estimated binary masks, the mixed amplitude spectrum is divided into human voice amplitude spectrum and musical voice amplitude spectrum. The human voice amplitude spectrum and the mixed phase spectrum are combined and then subjected to inverse short-time Fourier transform to reconstruct the human voice time-domain signal. The music amplitude spectrum and the mixed phase spectrum are combined and then subjected to inverse short-time Fourier transform to reconstruct the music time-domain signal, thereby separating the human voice and music in the mixed microphone signal.
2. The method for separating human voice and musical vocals using the neural network algorithm according to claim 1, characterized in that, Before performing a short-time Fourier transform on each of the hybrid microphone signals, the hybrid microphone signals are subjected to framing, windowing, and adjacent frame overlap processing, so that the hybrid microphone signals are represented as multiple time-frequency units; the windowing uses a smooth window function to reduce the influence of frame boundaries on the spectral representation; amplitude information and phase information are extracted from the time-frequency units respectively, the amplitude information is used as the input feature of the neural network, the phase information is retained as the phase reference, and the neural network input features are divided into feature blocks of uniform size for batch training and inference of the bidirectional long short-term memory neural network separation model.
3. The method for separating human voice and musical vocals using the neural network algorithm according to claim 1, characterized in that, The bidirectional long short-term memory (LSTM) neural network separation model includes a sequentially connected input layer, several bidirectional LSTM layers, a random deactivation regularization layer, a fully connected output layer, a nonlinear activation layer, and a norm regularization layer. The bidirectional LSTM layers include LSTM units that process the neural network input features forward in time and LSTM units that process the neural network input features backward in time, to simultaneously learn the temporal correlation of human voices and the temporal correlation of musical notes between consecutive time frames in the mixed microphone signal. The random deactivation regularization layer is used to suppress overfitting of the model to training samples. The fully connected output layer is used to map the temporal representation output by the bidirectional LSTM layers to the embedded representation of each time-frequency unit. The nonlinear activation layer is used to constrain the embedded representation to a value form that facilitates masking. The norm regularization layer is used to improve the stability of the embedded representation during the clustering stage.
4. The method for separating human voice and musical vocals using the neural network algorithm according to claim 3, characterized in that, The training objective of the bidirectional long short-term memory neural network separation model is ideal binary masking. Ideal binary masking is determined by comparing the relative dominance of the spectral amplitude of the human voice source and the spectral amplitude of the musical voice source within the same time-frequency unit in the training samples. When the spectral amplitude of the human voice source is dominant, the corresponding time-frequency unit is labeled as the human voice class; when the spectral amplitude of the musical voice source is dominant, the corresponding time-frequency unit is labeled as the musical voice class. During training, the loss is constructed based on the difference between the estimated affinity formed by the embedding representation output by the network and the target affinity formed by the ideal binary masking. An adaptive moment estimation optimization algorithm is used to update the model parameters through backpropagation, so that time-frequency units belonging to the same sound source category are closer to each other in the embedding space, and time-frequency units belonging to different sound source categories are farther apart in the embedding space.
5. The method for separating human voice and musical vocals using the neural network algorithm according to claim 4, characterized in that, The training samples are obtained by acquiring human voice and music audio from the training audio set, generating room impulse responses from the sound source location to each microphone location according to preset sound source direction and acoustic space parameters, and convolving the human voice and music audio with the corresponding room impulse responses to obtain single-source propagation signals received by each microphone. The single-source propagation signals are then superimposed into a mixed training signal within the same microphone channel according to different energy relationships. The room impulse responses are generated using the image sound source method based on the size of the enclosed acoustic space, the microphone array location, the sound source location, and the reverberation time. The amplitude spectrum is extracted from the mixed training signal as training input, and the ideal binary mask is generated from the human voice and music spectra that retain the category labels, thereby enabling the bidirectional long short-term memory neural network separation model to learn the ability to separate overlapping human voices and music under room reverberation conditions.
6. The method for separating human voice and musical vocals using a neural network algorithm according to claim 1, characterized in that, Clustering the embedded representation includes treating the embedded representation as sample points in the embedding space, using the mean clustering algorithm to divide the sample points into human voice clusters and musical voice clusters, and rearranging them according to the category labels obtained from the clustering into the estimated binary mask corresponding to the mixed amplitude spectrum; the masking part of the estimated binary mask corresponding to the human voice cluster is used to retain the time frequency units belonging to the human voice in the mixed amplitude spectrum, and the masking part of the estimated binary mask corresponding to the musical voice cluster is used to retain the time frequency units belonging to the musical voice in the mixed amplitude spectrum.
7. The method for separating human voice and musical vocals using a neural network algorithm according to claim 1, characterized in that, When the mixed amplitude spectrum is divided into a vocal amplitude spectrum and a musical amplitude spectrum based on the estimated binary mask, the vocal amplitude spectrum is obtained by combining the retained part corresponding to the estimated binary mask with the mixed amplitude spectrum unit by unit, and the musical amplitude spectrum is obtained by combining the complementary part corresponding to the estimated binary mask with the mixed amplitude spectrum unit by unit. The human voice amplitude spectrum and the music amplitude spectrum are respectively combined with the mixed phase spectrum to form a complex spectrum. Then, through inverse short-time Fourier transform and inter-frame overlap synthesis, the human voice time-domain signal and the music time-domain signal corresponding to each microphone channel are obtained.
8. The method for separating human voice and musical vocals using a neural network algorithm according to claim 1, characterized in that, After obtaining the human voice time-domain signal and the music time-domain signal from each microphone channel, the human voice multi-channel signal composed of the reconstructed signals from each human voice source and the music multi-channel signal composed of the reconstructed signals from each music source are respectively input into the pre-trained single-source localization model. The single-source localization model extracts sound intensity features and generalized cross-correlation phase transformation features from the human voice multi-channel signal and the music multi-channel signal, and represents the sound source direction as a discrete direction category through a convolutional neural network, thereby outputting the human voice source direction and the music source direction while separating the human voice and the music.
9. A vocal and instrumental separation system implemented using a neural network algorithm, characterized in that, include: The acquisition module is used to acquire a mixed microphone signal, which includes a human voice propagation component formed by the propagation of a human voice source through the corresponding room impulse response, a curved sound propagation component formed by the propagation of a curved sound source through the corresponding room impulse response, and a noise component. The feature transformation module is used to perform short-time Fourier transform on each of the hybrid microphone signals to obtain the hybrid amplitude spectrum and hybrid phase spectrum corresponding to the hybrid microphone signal, and to organize the hybrid amplitude spectrum into neural network input features according to time frames and frequency units; The feature transformation module is used to input the neural network input features into a trained bidirectional long short-term memory neural network separation model, and to perform temporal modeling on the neural network input features along the forward and reverse time directions through the bidirectional long short-term memory neural network separation model, and output the embedding representation corresponding to each time frequency unit; The clustering module is used to cluster the embedded representation to obtain estimated binary masks corresponding to human voice and musical voice classes, respectively, and to divide the mixed amplitude spectrum into human voice amplitude spectrum and musical voice amplitude spectrum based on the estimated binary masks. The separation module is used to combine the human voice amplitude spectrum with the mixed phase spectrum and then perform an inverse short-time Fourier transform to reconstruct the human voice time-domain signal, and to combine the music voice amplitude spectrum with the mixed phase spectrum and then perform an inverse short-time Fourier transform to reconstruct the music voice time-domain signal, thereby separating the human voice and music voice in the mixed microphone signal.