Adaptive frequency band optimization chorus sound separation method and system based on harmonic preservation

Abstract

The present application relates to a kind of adaptive band optimization chorus sound separation method and system based on harmonic maintenance, first, the signal is collected by microphone array and time delay compensation and multi-channel time-frequency transformation are carried out;Sound source spatial positioning and initial band division are carried out using MUSIC algorithm;Using the greedy optimization strategy, the band boundary is dynamically adjusted and optimized by maximizing the objective function including signal-to-noise ratio and harmonic integrity measure;Then, the fundamental frequency is obtained by harmonic analysis and detection, and the time-varying harmonic trajectory is tracked across frames by Viterbi algorithm;On this basis, combined with MVDR beam forming and the introduction of harmonic maintenance factor, enhanced time-frequency representation is obtained by soft time-frequency mask;Finally, inverse short-time Fourier transform time-domain reconstruction and synchronous harmonic phase are carried out.The present application realizes the accurate allocation of voice part spectrum, effectively reduces the overlap interference, protects the overtone column of human voice while accurately separating, significantly improves the timbre fidelity and naturalness of separated voice part.

Description

Technical Field

[0001] This invention relates to the field of digital signal processing technology, and in particular to an adaptive frequency band optimization method and system for separating choral vocal parts based on harmonic preservation. Background Technology

[0002] With the rapid development of digital music processing technology, choral voice separation technology has shown broad application prospects in music production, music education, and intelligent audio analysis. However, choral works typically contain dense harmonic progressions of multiple voices (such as soprano, mezzo-soprano, tenor, and bass) within the same or similar pitch range, resulting in severe spectral overlap between the voices in the time-frequency domain, and the unique harmonic overtone structure of the human voice intertwines. Existing audio separation technologies generally suffer from rigid frequency band allocation strategies, insufficient harmonic structure preservation, and severe inter-voice interference when facing such complex scenarios, making it difficult to meet the stringent requirements of professional music production for the fidelity of the separated voices' sound quality and timbre. Therefore, there is an urgent need for a choral voice separation method that can adaptively optimize frequency band allocation and effectively maintain harmonic characteristics.

[0003] I. End-to-end separation methods based on deep neural networks and their shortcomings Current mainstream methods employ U-Net, Conv-TasNet, or Transformer architectures to directly predict the time-frequency representation or time-domain waveform of a sound source through supervised learning. However, these methods suffer from the following major drawbacks: First, deep networks often excessively smooth spectral details during the separation process, leading to the weakening or distortion of the harmonic overtone series unique to choral parts, severely affecting timbre fidelity. Second, these methods are highly dependent on data, requiring a large number of labeled choral datasets (such as the ChoralSingingDataset) for training, while their generalization ability significantly decreases when facing different choirs and different musical styles in practical applications. Furthermore, the fixed convolutional kernels or receptive fields of the network are difficult to adaptively handle the overlapping spectral regions of different vocal ranges (such as soprano and bass).

[0004] II. Deficiencies of Nonnegative Matrix Factorization (NMF) and its Extended Methods Nonnegative matrix factorization (NMF) methods decompose the temporal spectrum into the product of the basis function matrix and the activation coefficient matrix, and utilize the sparsity constraint of the basis functions to achieve vocal separation. Extended algorithms include RNMF and SparseNMF. However, this type of method has obvious limitations: traditional NMF uses a fixed spectral template, which cannot adapt to the spectral deformation (formant variation) of the same voice part at different pitches in choral singing; when multiple voice parts are at the same or adjacent pitches (such as the octave overlap between soprano and mezzo-soprano), the linear superposition assumption of NMF leads to blurred separation boundaries; in addition, this method lacks a mechanism for tracking the evolution of harmonics over time, which easily produces "discontinuity" phenomena.

[0005] III. Blind Source Allocation (ICA / IVA) Based Methods and Their Limitations Blind source separation methods utilize the statistical independence assumption of sound source signals to separate mixed signals through higher-order cumulants or information maximization criteria, extending to the IVA method of multivariate analysis. However, in the scenario of choral voice separation, this type of method faces fundamental obstacles: choral voices are highly correlated in terms of rhythm and harmonic progression (such as parallel thirds and sixths), which seriously violates the statistical independence premise of ICA; the inconsistent arrangement order of frequency domain ICA in different frequency bands leads to a permutation problem of "misaligned splicing" in the spectrum of the separated voices; in addition, the separation results have amplitude ambiguity and cannot maintain the relative loudness balance of the original choral voices.

[0006] IV. Limitations of methods based on time-frequency masking and clustering Time-frequency masking-based methods assign time-frequency units to different vocal parts by calculating binary masks (BMMs) or soft masks (RatioMasks) and combining them with pitch tracking (such as the pYIN algorithm) or deep clustering. The drawbacks of this method are: in dense ensemble singing, errors in multi-pitch estimation directly lead to incorrect mask assignment, resulting in "vocal part leakage"; traditional methods, based on single-frame spectral peak detection, lack cross-frame harmonic continuity constraints, causing "vibrato" or "breaks" in the separated vocal parts; furthermore, using a fixed STFT window length makes it impossible to simultaneously optimize temporal resolution (for rapidly changing vocals) and frequency resolution (for harmonic structures).

[0007] V. Limitations of Methods Based on Computational Auditory Scene Analysis (CASA) Methods based on computational auditory scene analysis simulate the human auditory mechanism, utilizing cues such as harmonicity and temporal continuity to organize sound sources. However, traditional CASA employs a fixed Gammatone filter bank, which cannot be dynamically adjusted according to the actual frequency band distribution of choral parts, leading to the loss of high-frequency overtones or the introduction of low-frequency noise. When the frequency band is widened to maintain harmonic integrity, crosstalk from other parts is introduced, while strict separation destroys the harmonic structure, creating a contradiction between harmonic preservation and separation. Furthermore, this method lacks the adaptive capability to automatically adjust the separation strategy according to the style of the choral work (such as Baroque choral music versus modern pop choral music).

[0008] Patent application CN108198568A proposes a method and system for multi-source localization. The method includes: acquiring an initial set of speech signals; preprocessing each initial speech signal; dividing the preprocessed signal into multiple sub-bands using a cochlear basilar membrane filter; obtaining the cross-power spectral density function of the same sub-band; fusing the cross-power spectral density functions of all sub-bands; obtaining the two-dimensional position / fundamental frequency characteristic function of the microphone pair corresponding to each first speech signal pair; obtaining the weighted two-dimensional position / fundamental frequency characteristic function of each microphone pair; obtaining the fused weighted two-dimensional position / fundamental frequency characteristic function; and determining the azimuth and fundamental frequency of each sound source within a set threshold range based on the function value of the fused weighted two-dimensional position / fundamental frequency characteristic function. However, relying solely on fixed filter sub-band division for sound source localization and fundamental frequency estimation lacks adaptive spectrum optimization and cross-frame harmonic tracking mechanisms. It cannot eliminate overlapping interference in densely choral singing scenarios, nor can it achieve high-fidelity, distortion-free multi-part audio separation and reconstruction.

[0009] Based on the above five mainstream choral vocal separation techniques, existing methods have the following three common drawbacks: First, the frequency band allocation strategy is rigid. Whether it is the fixed convolution kernel of deep neural networks, the fixed basis function of nonnegative matrix factorization, the fixed frequency band filter for blind source separation, or the fixed STFT window length or Gammatone filter bank used in time-frequency masking and computational auditory scene analysis, all adopt a static and coarse-grained frequency band allocation strategy. It cannot adaptively adjust according to the dynamic distribution characteristics of choral parts in the time-frequency domain, which makes it difficult to achieve optimal allocation of spectrum resources when dealing with different vocal range combinations (such as the wide frequency span of soprano and bass) or pitch changes of the same vocal part.

[0010] Second, the ability to preserve harmonic structure is insufficient. Existing methods overemphasize the statistical independence or energy differences between sound sources during the separation process, neglecting the requirement for the integrity and continuity of the harmonic overtone series unique to the human voice. The spectral smoothing effect of deep networks, the linear approximation of matrix decomposition, and the hard decision mechanism of masking methods all cause attenuation or breakage of higher harmonic components, resulting in timbre distortion of the separated vocal parts and loss of the naturalness and fullness of choral vocals.

[0011] Third, there is severe interference between vocal parts in the overlapping frequency spectrum region. In the mid-frequency region of choral singing, the fundamental frequencies and overtones of multiple vocal parts highly overlap. Existing technologies lack effective harmonic-noise identification and tracking mechanisms, making it impossible to accurately distinguish harmonic components belonging to different vocal parts. This results in the widespread phenomenon of "vocal part leakage" and blurred separation boundaries. Summary of the Invention

[0012] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a harmonic-preserving adaptive frequency band optimization choral part separation method and system that uses adaptive frequency band optimization to accurately allocate the spectrum to reduce inter-voice overlap interference, and combines harmonic preservation mechanisms such as cross-frame harmonic tracking and phase synchronization to achieve accurate separation of multiple voices in close-knit choral singing, while effectively avoiding timbre distortion and significantly improving the naturalness and auditory quality of the separated voice signals.

[0013] The objective of this invention can be achieved through the following technical solutions: An adaptive frequency band optimization method for choral vocal separation based on harmonic preservation includes the following steps: Multi-channel chorus signals are acquired, and the generalized cross-correlation function between each pair of microphone signals is calculated to estimate the inter-channel delay. Based on the estimated inter-channel delay, time shift compensation is performed on the multi-channel chorus signals to obtain aligned multi-channel signals. Short-time Fourier transform is performed on the aligned multi-channel signal to obtain the multi-channel time-frequency representation, and the spatial covariance matrix is ​​calculated using the multi-channel time-frequency representation; Eigenvalue decomposition of spatial covariance matrix is ​​performed to obtain signal subspace and noise subspace. Multi-signal classification spatial spectrum is calculated to determine the direction of each sound part. Frequency band energy of each sound part is calculated and frequency band boundaries of each sound part are initialized. The spectral centroid and spectral spread of each voice part in the multi-channel time-frequency representation are extracted. A greedy optimization strategy based on boundary movement is adopted. The frequency band boundary is adjusted by maximizing the objective function that considers the signal-to-noise ratio, the inverse of the spectral overlap, and the harmonic integrity metric. A filter bank is constructed based on the adjusted frequency band boundary to selectively filter the frequency band of each voice part and obtain the time-frequency representation after frequency band filtering. Harmonic analysis and function detection are used to detect the fundamental frequency of each voice part. An ideal harmonic template is established using the detected fundamental frequency to mark the theoretical distribution position of the harmonic components of each voice part. The Viterbi algorithm is used to track the fundamental frequency trajectory and amplitude changes of each harmonic across frames to obtain the time-varying harmonic trajectory. A beamformer is constructed using the guide vectors of each sound part to perform spatial filtering on the time-frequency representation after the frequency band filtering to obtain a preliminary separated time-frequency representation. A soft time-frequency mask with a harmonic preservation factor is constructed based on the time-varying harmonic trajectory. The soft time-frequency mask is applied to the preliminary separated time-frequency representation and the enhanced time-frequency representation is obtained by smoothing in the harmonic frequency direction. The enhanced time-frequency representation is converted into a separated time-domain signal by inverse short-time Fourier transform. The fundamental frequency phase of the separated time-domain signal is extracted and the phases of each harmonic component are adjusted to align with the ideal harmonic phase to obtain the target separated sound part signal.

[0014] Furthermore, the multi-channel chorus signal is acquired by microphones arranged at the four vertices of a three-dimensional coordinate system. The first microphone is located at the origin of the three-dimensional coordinate system, the second microphone is located at a set distance in the positive direction of the horizontal axis, the third microphone is located at a set distance in the positive direction of the vertical axis, and the fourth microphone is located at a set distance in the positive direction of the vertical axis.

[0015] Furthermore, the generalized cross-correlation function is calculated using the following formula: in, Indicates the first The microphone and the first The relative time delay between microphones, in seconds; This represents the integral variable of the time delay, i.e., the candidate time delay value, in seconds; Indicates the first Frequency domain representation of the microphone signal; Indicates the first Frequency domain conjugate representation of each microphone signal; The magnitude of the cross-power spectrum is represented by a non-negative real number. This represents the frequency integral variable, with the unit being Hz; Represents the imaginary unit; The power represents a complex exponential function, i.e., a phase rotation factor; This represents the time delay value that maximizes the integral. The operator.

[0016] Furthermore, the formula for calculating the objective function is as follows: in, This represents the objective function for frequency band optimization. Indicates the total number of voice parts; This represents the voice part index, with values ​​1, 2, ... ; Indicates the first Number of sub-bands per voice part; This represents the sub-band index, with values ​​1, 2, ... ; Indicates the first The first voice part in the The signal-to-noise ratio of each subband; Indicates the first The first voice part in the Spectral overlap between individual sub-bands and other sound parts; This represents the harmonic integrity measure of the s-th voice part in the m-th subband, satisfying 0 ≤ ≤1; , , Let represent the weighting coefficient, which is a non-negative real number, satisfying . It equals 1.

[0017] Furthermore, the formula for calculating the spectral overlap is as follows: in, For the first The first voice part in the Spectral overlap of each subband For indexing competing voice parts, Index the start and end frequency points covered by this sub-band. ) is the first Each voice part at the time frequency point The complex time-frequency amplitude, For the time-frequency amplitude of the competing voice parts, To prevent overflow of constants.

[0018] Furthermore, the harmonic analysis and function are as follows: in, Harmonic analysis and functions at candidate fundamental frequencies The value at; It is a candidate fundamental frequency; It is the harmonic order; It is the highest harmonic order considered; The mixed signal is in the first... Frame, frequency Time-frequency representation at a location; It is the amplitude of that frequency component; It is the first Attenuation weight of subharmonics.

[0019] Furthermore, the harmonic preservation factor in the soft time-frequency mask is assigned a value based on the set to which the frequency point belongs, as follows: in, Harmonic preservation factor For harmonic enhancement coefficient, This represents the conflict zone suppression coefficient. For the first The first voice part in the n The set of harmonic frequency indexes of a frame. For the first The first voice part in the n A set of indexes for noise / clear audio points in a frame.

[0020] Furthermore, the smoothing in the harmonic frequency direction involves calculating the sum of the products of the interpolation weights of each harmonic and the original time-frequency value within the upper and lower limits of the target harmonic order, and then dividing the sum of the products by the sum of the interpolation weights to obtain the enhanced time-frequency value.

[0021] Further, after acquiring the target separated acoustic signal, the harmonic distortion rate and the acoustic leakage suppression ratio are calculated. The harmonic distortion rate is calculated based on the square of the absolute value of the difference between the extracted harmonic amplitude and the ideal harmonic amplitude. The acoustic leakage suppression ratio is calculated based on the square of the absolute value of the convolution between the target separated acoustic signal and the competing acoustic reference signal. If either the harmonic distortion rate or the acoustic leakage suppression ratio is greater than a preset threshold, at least one of the harmonic enhancement coefficients in the frequency band boundary and the soft time-frequency mask is readjusted until the recalculated harmonic distortion rate is not greater than 0.1 and the recalculated acoustic leakage suppression ratio is not greater than -18dB.

[0022] An adaptive frequency band optimization choral vocal separation system based on harmonic preservation, comprising: The acquisition and compensation module is configured to acquire multi-channel chorus signals, calculate the generalized cross-correlation function between each pair of microphone signals to estimate the inter-channel delay, and perform time-shift compensation on the multi-channel chorus signals based on the estimated inter-channel delay to obtain aligned multi-channel signals. The transformation and statistics module is configured to perform short-time Fourier transform on the aligned multi-channel signal to obtain a multi-channel time-frequency representation, and use the multi-channel time-frequency representation to calculate the spatial covariance matrix; The spatial positioning module is configured to perform eigenvalue decomposition on the spatial covariance matrix to obtain the signal subspace and noise subspace, calculate the multi-signal classification spatial spectrum to determine the direction of each sound part, calculate the frequency band energy of each sound part, and initialize the frequency band boundary of each sound part. The frequency band optimization module is configured to extract the spectral centroid and spectral spread of each sound part in the multi-channel time-frequency representation, adjust the frequency band boundary by maximizing the objective function that includes the signal-to-noise ratio, the inverse of the spectral overlap, and the harmonic integrity metric, and construct a filter bank based on the adjusted frequency band boundary. The harmonic tracking module is configured to use harmonic analysis and functions to detect the fundamental frequency of each sound part, establish an ideal harmonic template using the detected fundamental frequency, and use the Viterbi algorithm to track the fundamental frequency and harmonic amplitude across frames to obtain the time-varying harmonic trajectory. The separation and enhancement module is configured to use the guide vectors of each sound part to construct a beamformer to perform spatial filtering on the multi-channel time-frequency representation to obtain a preliminary separated time-frequency representation. Based on the time-varying harmonic trajectory, a soft time-frequency mask with a harmonic preservation factor is constructed. The soft time-frequency mask is applied to the preliminary separated time-frequency representation and the enhanced time-frequency representation is obtained by smoothing in the harmonic frequency direction. The reconstruction synchronization module is configured to perform inverse short-time Fourier transform on the enhanced time-frequency representation to convert it into a separated time-domain signal, extract the fundamental frequency phase of the separated time-domain signal, and adjust the phases of each harmonic component to align with the ideal harmonic phase to obtain the target separated sound part signal.

[0023] Compared with the prior art, the present invention has the following beneficial effects: 1. Accurately align multi-channel signals and correct time delay errors caused by spatial sound wave propagation: Acquire multi-channel chorus signals, calculate the generalized cross-correlation function between each pair of microphone signals to estimate the inter-channel time delay, and perform time shift compensation on the multi-channel chorus signals based on the estimated inter-channel time delay to obtain aligned multi-channel signals; 2. Achieve high-resolution sound field spatial localization and multi-directional sound source differentiation: Perform short-time Fourier transform on the aligned multi-channel signals to obtain multi-channel time-frequency representation, use the multi-channel time-frequency representation to calculate the spatial covariance matrix, perform eigenvalue decomposition on the spatial covariance matrix to obtain the signal subspace and noise subspace, and calculate the multi-signal classification spatial spectrum to determine the direction of each sound part; 3. Break through the rigid limitations of fixed frequency band division, achieve adaptive and accurate spectrum allocation and minimize inter-voice overlap interference: calculate the frequency band energy of each voice and initialize the frequency band boundary of each voice, extract the spectral centroid and spectral spread of each voice in the multi-channel time-frequency representation, adopt a greedy optimization strategy based on boundary movement, adjust the frequency band boundary by maximizing the objective function that includes signal-to-noise ratio, the inverse of spectral overlap and harmonic integrity measure, and construct a filter bank based on the adjusted frequency band boundary; 4. Overcome the ambiguity and jumps in single-frame detection and ensure the continuity and stability of the human voice harmonic structure in time sequence: use harmonic analysis and function detection to detect the fundamental frequency of each voice part, use the detected fundamental frequency to establish an ideal harmonic template, and use the Viterbi algorithm to track the fundamental frequency and harmonic amplitude across frames to obtain the time-varying harmonic trajectory. 5. While suppressing interference, it effectively protects the vocal overtone series from being destroyed, and significantly improves the sound quality and timbre fidelity of the separated voice parts: a beamformer is constructed using the guide vectors of each voice part to perform spatial filtering on the multi-channel time-frequency representation to obtain the initial separated time-frequency representation. A soft time-frequency mask with a harmonic preservation factor is constructed based on the time-varying harmonic trajectory. The soft time-frequency mask is applied to the initial separated time-frequency representation and the enhanced time-frequency representation is obtained by smoothing in the harmonic frequency direction. 6. To avoid timbre distortion such as "metallic sound" caused by phase misalignment during reconstruction and to maintain the cohesion and naturalness of the choral vocals: the enhanced time-frequency representation is converted into a separated time-domain signal by inverse short-time Fourier transform, the fundamental frequency phase of the separated time-domain signal is extracted, and the phases of each harmonic component are adjusted to align with the ideal harmonic phase to obtain the target separated vocal part signal. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system module block diagram of the present invention. Detailed Implementation

[0025] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0026] Example 1 This embodiment describes in detail the adaptive frequency band optimization choral vocal separation method based on harmonic preservation, such as... Figure 1 As shown, the specific implementation method is as follows: S1: Spatial Acquisition and Delay Compensation Using a Four-Microphone Tetrahedral Array The primary challenge in separating choral voices lies in accurately acquiring the three-dimensional spatial information of the sound field. Traditional single microphones can only record the time changes in sound pressure. While dual microphones or linear arrays can distinguish left from right, they suffer from "front-back blurring" (unable to distinguish between front and rear sound sources) and have limited resolution for multiple sound sources on the same horizontal plane. More importantly, the wavelength of sound waves (17cm to 6.8m) corresponding to the fundamental frequency range of human voices (50-2000Hz) is roughly equivalent to the distance between microphones. If the array geometry is not properly designed, it can lead to phase aliasing, making it impossible to accurately estimate the direction. This embodiment uses a four-microphone array with a tetrahedral geometry, utilizing the time difference (phase difference) of sound waves arriving at microphones at different positions to provide complete three-dimensional spatial sampling for subsequent sound source localization.

[0027] Four microphones are positioned at the four vertices of a three-dimensional coordinate system: the first microphone M1 is located at the origin (0, 0, 0) as the reference point; the second microphone M2 is located at (d, 0, 0); the third microphone M3 is located at (0, d, 0); and the fourth microphone M4 is located at (0, 0, d). This non-coplanar arrangement ensures that incident sound waves from any direction produce linearly independent phase responses at the four microphones. The choice of spacing d is crucial: if d is too small (less than 5 cm), the phase difference of low-frequency sound waves is less than the resolvable limit; if d is too large (greater than 20 cm), high-frequency sound waves will exhibit phase entanglement (phase difference exceeding 2). (This leads to directional ambiguity). In this embodiment, d is set to 8 to 15 cm (corresponding to half to full wavelength of the fundamental frequency of human voice), which can provide a resolvable and unique phase difference throughout the entire audio band.

[0028] Four microphones are triggered by the same clock source, ensuring a sampling synchronization error of less than 1 microsecond. Sampling rate Choosing 48000Hz satisfies the Nyquist sampling theorem (the highest frequency of human voice is approximately 8kHz) while also reserving sufficient bandwidth for subsequent processing. A 24-bit sampling depth and a dynamic range of 144dB are sufficient to capture the significant differences between loud and soft parts in a chorus. The four raw signals acquired are denoted as follows: ,in This represents a continuous time variable, with the unit being seconds.

[0029] Because sound waves travel at a limited speed in air (approximately 343 m / s at room temperature), there is a slight time difference (microseconds to milliseconds) between sounds from the same source reaching different microphones. For example, for a distance d of 10 cm, the maximum time difference is approximately... Without compensation, these delays would cause subsequent spatial processing to fail completely. This embodiment uses a generalized cross-correlation phase transform to estimate the delay. The core idea is to use the phase information (rather than the amplitude information) of the cross-power spectrum for delay estimation because phase is more robust to room reverberation and background noise. For the signals from microphones i and j, their cross-power spectra are first calculated. ,in yes Fourier transform, yes The complex conjugate of the Fourier transform. Then divide by the absolute value of the amplitude of the cross-power spectrum. This is equivalent to retaining only the phase information and normalizing the amplitude of all frequency components to 1, thereby suppressing noise and reverberation peaks in the amplitude spectrum.

[0030] The generalized cross-correlation function is obtained through the inverse Fourier transform: in, Indicates the first The microphone and the first The relative time delay between microphones, in seconds; This represents the integral variable of the time delay, i.e., the candidate time delay value, in seconds; Indicates the first The frequency domain representation of a microphone signal is a complex number; Indicates the first The frequency domain conjugate representation of a microphone signal is a complex number; The magnitude of the cross-power spectrum is represented by a non-negative real number. This represents the frequency integral variable, with the unit being Hz; Represents the imaginary unit; The power represents a complex exponential function, i.e., a phase rotation factor; This represents the time delay value that maximizes the integral. The operator.

[0031] The function exhibits a sharp peak at the actual time delay. After obtaining the time delay, time shift compensation is performed on each signal; the compensation formula is as follows: .in Indicates the first The time-domain signal from each microphone after time delay compensation; Indicates the first Each microphone relative to the reference microphone The delay is measured in seconds. This represents the time variable, in seconds. After compensation, the sound waves from the reference direction are time-aligned across the four channels, resulting in... This lays the foundation for subsequent processing.

[0032] S2: Time-Frequency Transformation and Spatial Statistical Characteristic Extraction Speech and music signals are non-stationary (frequency components change over time). Traditional Fourier transforms assume the signal is infinitely long and stationary, making them unsuitable for analyzing time-varying signals. Furthermore, to utilize the multi-channel spatial information obtained from S1, it is necessary to construct statistics that describe the spatial structure of the sound field. This step uses Short-Time Fourier Transform (STFT) to decompose the signal into a time-frequency representation and constructs a spatial covariance matrix to characterize the statistical relationships between the multiple channels.

[0033] S201: Short-Time Fourier Transform (STFT) The time-domain signal is divided into overlapping short frames, and the signal is approximated as stationary within each frame. Specific parameters are set as follows: frame length N is 2048 samples (approximately 42.7ms), which can capture 2-3 cycles of the fundamental frequency of human voice (assuming a fundamental frequency of 50-200Hz); frame shift H is 512 samples (75% overlap rate) to ensure time continuity; a Hanning window w(m) is added to reduce spectral leakage (spectral spread caused by truncation effect).

[0034] Transformation formula: in, It is the first The microphone signal at the first n Frame, complex time-frequency representation of the k-th frequency point (complex number); The frame index (a non-negative integer representing the time frame number); It is a frequency point index (integer, 0≤ < N (representing the discrete frequency index). It is the STFT frame length (positive integer, number of samples, 2048 in this embodiment); It is the sample index (integer, 0≤m <N (representing the intra-frame sample number). It is the time-domain signal value (real number) of the i-th microphone at the m-th sample point in the n-th frame; It is the weight of the Hanning window function at sample m (a real number). 0≤w(m)≤1 ) It is the frame shift (a positive integer, the number of samples, which is 512 in this embodiment); It is the complex exponential basis function (complex number) of the Discrete Fourier Transform. This representation also preserves time information (frame index). ) and frequency information (frequency point) This forms a two-dimensional time-frequency diagram, which is suitable for analyzing time-varying signals.

[0035] S202: Calculation of Spatial Covariance Matrix At each time-frequency point (n, k), a spatial covariance matrix is ​​constructed using the four-channel signal, expressed as: in, It is the spatial covariance matrix (a 4x4 Hermitian matrix with complex elements). = It is a four-channel time-frequency vector (a 4x1 column vector, with elements representing the time-frequency of each microphone). It is a vector Hermitian transpose (a 1x4 row vector); matrix elements , representing the cross-correlation between the i-th and j-th microphone signals at that time-frequency point, where express .

[0036] The spatial covariance matrix describes the spatial correlation of the sound field at that time-frequency point: diagonal elements )= It is the first i The off-diagonal element of the microphone's self-power (signal strength, a non-negative real number). = It is the first i With the j The cross-correlation of the individual microphones, including the phase angle of the cross-correlation of amplitude and phase information. It directly reflects the phase difference of sound waves arriving at the two microphones, thus indicating the direction of the sound source. For example, if the sound source is at... On the perpendicular bisector of the line connecting them, then and Same phase, It is a positive real number; if the sound source is close to ,but Phase advance , It is a complex number, and the phase angle is positive.

[0037] This matrix forms the mathematical basis for subsequent MUSIC spatial spectrum estimation and MVDR beamforming.

[0038] S3: Sound source spatial localization and initial frequency band division After obtaining the spatial statistical characteristics of the sound field, two key issues need to be addressed: first, the direction from which each sound source originates, and second, the frequency bands in which their energy is distributed. Traditional energy thresholding methods cannot distinguish multiple sound sources within the same frequency band, while fixed frequency band division cannot adapt to the dynamic spectrum of the sound sources. This step utilizes subspace decomposition for high-resolution direction finding and adaptively initializes the frequency bands based on the energy distribution.

[0039] S301: MUSIC Spatial Spectrum Estimation The MUSIC (MultipleSignalClassification) algorithm is based on the following physical facts: sound signals from different directions are statistically approximately independent, spanning a low-dimensional signal subspace; while noise is uniformly distributed in all directions, spanning a noise subspace. These two subspaces are orthogonal, meaning that any vector in the signal subspace is orthogonal to any vector in the noise subspace. The algorithm separates these two subspaces by performing eigenvalue decomposition on the spatial covariance matrix. Let the eigenvalues ​​of the covariance matrix R be λ1≥λ2≥...≥λ4, where the first S large eigenvalues ​​correspond to the signal subspace (S is the number of sound sources), and the remaining 4-S small eigenvalues ​​correspond to the noise subspace.

[0040] For a certain candidate direction Calculate its guiding vector This vector represents the phase response of the four microphones when a plane wave is incident from this direction, expressed as: in, It is a guiding vector (4x1 complex vector). Azimuth (the angle in the horizontal direction of the sound source, in radians or degrees). ); Pitch angle (angle perpendicular to the sound source, unit: radians or degrees) ); It is angular frequency ( (Unit: rad / s) It is the propagation delay of the sound wave from the reference point to the i-th microphone (unit: seconds).

[0041] Calculate the projection of the steering vector onto the noise subspace. in, It is the MUSIC spatial spectrum (a scalar representing direction). The estimated signal power (real number). It is the voice subspace matrix Matrix, by The eigenvectors are composed of the eigenvalues ​​corresponding to the smallest eigenvalues. This represents the number of sound sources. It is the conjugate transpose of the noise subspace ( matrix); It is the conjugate transpose of the steering vector (a 1×4 row vector), and the product in the denominator is a scalar, which is the square of the complex modulus. If there is a signal in this direction, the steering vector should fall into the signal subspace, be approximately orthogonal to the noise subspace, and have a very small projection value, leading to... A sharp peak appears; conversely, if there is no signal in that direction, the steering vector mainly falls into the noise subspace, resulting in a larger projection value. Smaller. (Scanned across the entire space) By drawing a spatial spectrum and detecting the position of the spectral peaks, the location can be determined. The MUSIC algorithm breaks through the Rayleigh limit (the resolution limitation of traditional beamforming), and when the signal-to-noise ratio is sufficient, the angular resolution can reach within 1 degree, which is enough to distinguish different voices that are several degrees apart on the stage.

[0042] S302: Initial Frequency Band Allocation Calculate the energy of each sound band: in, It is the first The energy of each sound part in the k-band ( (a real number, non-negative). It is the voice part index (positive integer, ). It is the total number of voice parts (a positive integer); It is the first Spatial filtering weights for each voice part (complex number, generated by beamformer); For reference microphone (e.g.) The time-frequency representation of (complex number); It is a frame index (summing and traversing all time frames); Initialize the set of frequency band boundaries ,in For the first One boundary frequency ( (Unit: Hz) .

[0043] S4: Adaptive Bandwidth Optimization and Dynamic Boundary Adjustment Traditional methods use fixed filter banks (such as uniform subbands or Mel frequency subbands), but the spectrum of choral parts is dynamic: the fundamental frequency of the same part differs at different pitches, and the spectra of different parts may overlap or separate. Fixed frequency bands cannot adapt to this dynamism, resulting in either excessively wide bands introducing interference or excessively narrow bands losing information. One of the core innovations of this embodiment is adaptive frequency band optimization: based on the real-time spectral characteristics of the current part, the center frequency and bandwidth of the subbands are dynamically adjusted so that each subband contains as much dominant energy as possible from one part, while maintaining the integrity of the harmonic structure.

[0044] S401: Analysis of the Spectral Characteristics of the Acoustic Parts Extract the spectral centroid of each voice part and spectral diffusivity : in, It is the spectral centroid of the s-th voice part in the nth frame (representing the energy concentration frequency position, a dimensionless frequency index, a real number); It is the spectral spread of the s-th voice part in the nth frame (representing the discreteness of energy distribution, a real, non-negative number); It is the time-frequency representation (complex number) of the s-th voice part after initial separation; It is a frequency index (summing and traversing all frequency points); It is the power spectral density value in time-frequency representation (real number, non-negative); It is the square of the deviation between the frequency point and the centroid.

[0045] Spectral centroid This indicates the frequency of energy concentration, reflecting the current pitch of the vocal part; the center of mass is higher for sopranos and lower for basses. Spectral dispersion. It represents the spectral bandwidth, reflecting the brightness of the timbre. Bright sounds have a large diffusion, while deep sounds have a small diffusion.

[0046] S402: Dynamic Band Boundary Adjustment and Objective Function Optimization Establish the objective function for frequency band optimization: in, denoted as the objective function for frequency band optimization, is a scalar and a real number; the larger the value, the better the separation effect. This represents the total number of voice parts and is a positive integer. This represents the voice part index, with values ​​1, 2, ... ; Indicates the first The number of sub-bands for each voice part is a positive integer. They can be different; This represents the sub-band index, with values ​​1, 2, ... ; Indicates the first The first voice part in the The signal-to-noise ratio of each subband is a real number and is non-negative; Indicates the first The first voice part in the The spectral overlap between each sub-band and other voice parts is dimensionless. ≥1 is a real number; This represents the harmonic integrity measure of the s-th voice part in the m-th subband, satisfying 0 ≤ ≤1, is a real number; , , Let represent the weighting coefficient, which is a non-negative real number, satisfying . Equal to 1; 1 divided by This represents the reciprocal of the overlap degree; the smaller the overlap, the larger this value. Optimization objectives include maximizing the signal-to-noise ratio to ensure signal clarity within each sub-band, minimizing acoustic overlap to reduce interference between different acoustic parts within the same sub-band, and maximizing harmonic integrity to maintain the continuity of the harmonic structure. A greedy algorithm is used to solve for the optimal set of frequency band boundaries. ,in For the optimized first One boundary frequency ( (Unit: Hz) This represents the total number of subbands after optimization.

[0047] (1) Spectral overlap Calculation formula For the The first of the voice parts Each sub-band has a frequency range of (Corresponding frequency index) arrive The overlap coefficient is defined as the ratio of the target sound energy to the total energy of other competing sound parts within the sub-band. However, to characterize the degree of overlap, the reciprocal of the competing energy ratio is usually calculated directly. Here, the overlap coefficient is used. Among them For the first The first voice part in the Spectral overlap of subbands (dimensionless) ≥1, real number). Index for competing voice parts ( and ), This is the start and end frequency index (integer) for the subband coverage. ) is the first Each voice part at the time frequency point The complex time-frequency amplitude (from the initial separation, such as the output of step S3). For the time-frequency amplitude of the competing voices, For a very small positive number (such as...) ( ), to prevent the denominator from being zero.

[0048] Physical meaning: When the energy of other voice parts is much less than that of the target voice part, the second term approaches 0. ≈1 indicates low overlap (ideal case); when the energy of other voice parts is comparable to the energy of the target voice part, ≈2; When the energy of other voice parts is much greater than that of the target voice part, A large value indicates that the sub-band is severely interfered with, which is detrimental to the separation of the target acoustic component. In the objective function... In the middle, we use 1 / As a reward, the smaller the overlap, the greater the contribution.

[0049] (2) Harmonic integrity measurement Calculation formula Harmonic integrity measurement measures the first The first voice part in the Whether a natural harmonic structure can be maintained within a sub-band (i.e., whether the energy at integer multiples of the fundamental frequency is completely preserved). The definition is as follows: in It is the first The set of frequency points near the subharmonics (the range of which is determined by the harmonic tracking results in S5) is as follows: in, It is a harmonic integrity measure (real number, 0 ≤ ≤1); K is the harmonic order (traversing all harmonics within this sub-band range); K is the harmonic number. Frequency index for subharmonic attribution (width 2) +1); It is the harmonic bandwidth half-width, usually taken as 2 to 3; : No. The fundamental frequency (Hz) of each voice part in the current frame.

[0050] Physical meaning: The closer the value is to 1, the more concentrated the energy within the sub-band is in harmonic components, indicating a complete harmonic structure; conversely, a smaller value indicates a higher proportion of non-harmonic noise or other acoustic interference within the sub-band, resulting in poor harmonic integrity. This index is used to constrain frequency band division to avoid "shredding" the harmonic structure, thereby maximizing harmonic integrity. In item.

[0051] Greedy search strategy Since the number of sub-band boundaries is usually small (e.g., the entire frequency band is divided into 20-40 sub-bands), a greedy algorithm or dynamic programming can be used to solve the problem. Here, we present an efficient greedy optimization strategy based on boundary movement: Input: Initial set of frequency band boundaries in Each sub-band initial bandwidth They may be equal or based on the Mel scale.

[0052] Optimization goal: Maximize Notice It changes with boundary variations (when some subbands may not contain the effective energy of a certain voice part). It will be less than the total number of subbands. M However, for simplification, a unified order can be used. M fixed).

[0053] Greedy iteration steps: 1. Initialization: Set the current boundary set. Calculate the current objective function value

[0054] 2. Iterative process (repeated until convergence or the maximum number of iterations is reached): For each boundary... Try to Move to the left (A is the frequency step size, such as 20Hz or 1 frequency index), resulting in a new boundary set. Then try to Move to the right ,get Then calculate separately. and Of all the attempts, choose to make Increase the maximum effective movement (if it exists) > (Movement), then update If no movement can improve If the algorithm converges, then the optimal boundary set is output. .

[0055] Precautions: Movement step size The size can be dynamically reduced (e.g., starting with a large step size and then decreasing it later) to improve efficiency. To prevent subbands from becoming too wide or too narrow, constraints can be added. For multi-frame scenes, each frame can be optimized independently; or a sliding window can be used to smooth boundary changes and avoid drastic jitter.

[0056] S403: Band Selectivity Processing and Adaptive Filter Bank Construction Based on the optimized frequency band boundaries, a reconfigurable filter bank is constructed: in, It is the first s The adaptive filter frequency response function (complex or real number) for each voice part. For continuous frequency variables (unit: Hz), ), It is the first s Number of sub-bands for each voice part (positive integer). index for sub-band （ ）。 yes No. s The first voice part in the The gain coefficient of each subband (a real number, adaptively adjusted according to the energy proportion of the sound part in that frequency band). It is the first The center frequency of each sub-band (unit: Hz). It is the first The bandwidth of each subband (unit: Hz). It is a rectangular window function (1 when the absolute value of the independent variable is ≤0.5, otherwise 0). It is the normalized frequency deviation (dimensionless).

[0057] This "selective filtering" ensures that each voice only passes through its dominant frequency band while suppressing other frequency bands. Experiments show that adaptive band optimization can improve separation by 15-20% compared to a fixed Mel filter bank, especially in the region with severe mid-frequency overlap.

[0058] S5: Harmonic Structure Detection, Modeling, and Cross-Frame Tracking Human voice is a harmonic-rich signal, with peaks (harmonics or overtones) in its spectrum at integer multiples of the fundamental frequency. This harmonic structure is the essential characteristic that distinguishes human voice from noise and musical instruments. Accurately detecting and tracking the harmonic structure is crucial for maintaining the "naturalness" of the separated sound. Traditional methods often focus only on the fundamental frequency, neglecting higher harmonics, resulting in a thin and distorted sound after separation. This step establishes an ideal harmonic template through harmonic analysis and fundamental frequency detection, and uses time-series tracking to maintain harmonic continuity.

[0059] S501: Multi-Fundamental Frequency Detection (Harmonic Analysis and...) Using harmonic analysis and (Harmonic Summation) methods: in, Harmonic analysis and functions at candidate fundamental frequencies The value at that location (scalar, non-negative real number, the larger the value, the higher the probability of that fundamental frequency). It is the candidate fundamental frequency (unit: Hz, positive real number). It is the harmonic order (positive integer). ). It is the highest harmonic order considered (a positive integer, usually taken as 8-15). The mixed signal is in the first... Frame, frequency The time-frequency representation at a point (complex number). It is the amplitude of the frequency component (a non-negative real number). It is the first The attenuation weight of the subharmonic (as...) Increase and decrease, usually (or exponential decay, positive real number) If candidate frequency If it is the fundamental frequency, then it is an integer multiple thereof. There will be harmonic peaks at certain points, and the cumulative sum can be very large; if it is a non-fundamental frequency, then... There are no peak values, and the cumulative sum is very small. By searching for peak values, the fundamental frequency of each voice part can be accurately estimated, even if multiple voice parts are playing simultaneously.

[0060] S502: Harmonic Overtone Series Modeling Based on the detected base frequency (No. (fundamental frequency of each voice part), establish an ideal harmonic template: in, It is the first The first voice part in the Ideal harmonic template value (non-negative real number) for a frequency point. For voice part index ( ). It is the first The first voice part Amplitude estimate of the subharmonic (non-negative real number). It is the highest order (positive integer) of the harmonics considered. It is the harmonic order ( ） . It is the Kroneckerdelta function (the function value is 1 when the independent variable is 0, otherwise it is 0). It is a rounding function that maps a real number to the nearest integer. It is the first The fundamental frequency of each voice part (unit: Hz) It is the STFT frequency resolution （ (Unit: Hz). It is the first The frequency index (real number) corresponding to the subharmonic. It is a discrete frequency point index (integer, ） This template serves as "prior knowledge," marking the theoretical positions of all harmonics in that voice part and guiding subsequent separation processing.

[0061] S503: Cross-frame harmonic tracking (Viterbi algorithm) The human voice is continuously changing, with the fundamental frequency fluctuating up and down with the melody. The Viterbi algorithm is used to track harmonic parameters in time. State vector: in, It is the first The first voice part in the The state vector of a frame ( (column vector). It is the sth voice part in the th... The fundamental frequency estimate of the frame (unit: Hz, positive real number). It is the first Frame number The observed amplitude of the subharmonic ( (a non-negative real number). It is the highest harmonic order considered. It is the vector transpose symbol.

[0062] Temporal smoothing employs first-order recursion: in, It is the first Frame number Amplitude estimate of the smoothed subharmonic (non-negative real number). It is the first Amplitude estimate (non-negative real number) after smoothing of the previous frame. It is the first Frame number The observed amplitude of the subharmonic (a non-negative real number). It is the time smoothing coefficient ( It is usually taken as 0.7-0.9 (a real number).

[0063] The tracking results form time-varying harmonic trajectories. ,in For the 8th voice part The second harmonic in the first Frame frequency (unit: Hz, ideally) This resolves the ambiguity of single-frame detection (such as octave error, which misclassifies the fundamental frequency as twice or half), ensuring the temporal continuity of the harmonic structure.

[0064] Fundamental state transition cost function in the Viterbi algorithm To achieve continuous and smooth fundamental frequency tracking, the Viterbi algorithm maintains a set of candidate fundamental frequency states for each frame. Starting from the state of the previous frame... Transition to the current frame candidate state The cost (negative log-likelihood) is defined as: in, As a price for fundamental frequency changes, This is the cost of changes in harmonic amplitude.

[0065] 1. Cost of fundamental frequency change in, It is the fundamental frequency transfer weighting coefficient (a positive real number that controls the smoothness, usually taken as 1.0~5.0). () is the candidate fundamental frequency (unit: Hz) of the previous frame and the current frame. It is the maximum allowable fundamental frequency variation per frame (unit: Hz), which is set according to the type of voice part. It is the norm exponent (usually 2 or 1). The cost function penalizes excessive inter-frame fundamental frequency jumps to prevent unnatural abrupt changes in tracking results. When the frequency change exceeds Afmax, the transition probability is zero (the cost is infinite), eliminating impossible state transitions.

[0066] 2. Cost of Harmonic Amplitude Variation in, This is the amplitude variation weighting coefficient (usually taken as 0.5~2.0). It is the harmonic amplitude after smoothing in the previous frame (already obtained in the time domain smoothing). It is the harmonic amplitude observation value of the current frame candidate state. It is a small constant to prevent division by zero.

[0067] The harmonic amplitudes of adjacent frames should not fluctuate drastically; this cost term ensures that the tracked amplitude changes are consistent with the smoothed historical values, avoiding amplitude "glitch" caused by noise.

[0068] 3. Special handling for "glissando" (adaptive coefficient adjustment) For time periods where rapid frequency changes are detected (e.g., via spectral centroid differential detection in step S4), the adjustment can be dynamically adjusted. and : Reduce smoothing strength to ; Relaxing the restrictions on change ; 4. Viterbi recurrence relation remember For the first Frames in baseband The minimum cumulative cost at the end is then calculated recursively as follows: Among them, the observation likelihood cost Harmonic analysis and Taking the negative logarithm yields (high SHs correspond to low costs).

[0069] S6: Harmonic Preservation-Based Acoustic Separation and Enhancement This is the core separation step in this embodiment. Traditional methods (such as spectral subtraction and independent component analysis) only focus on "separating the sound parts" and ignore "the sound quality after separation", which often results in the separated sound losing the brightness and fullness unique to human voices (because the harmonics are destroyed).

[0070] This embodiment proposes a harmonic preservation separation method: first, spatial filtering is performed using an MVDR beamformer (separating sound parts from different directions); then, guided by the harmonic structure, a harmonic constraint mask is applied to the separated time-frequency representation, and the harmonic amplitude is continuously enhanced. The entire process ensures that the separated sound parts still have a complete harmonic structure, resulting in a natural and full sound.

[0071] S601: MVDR Spatial Filtering Refinement Using the sound source direction information obtained in step S3, spatial filtering is performed using an MVDR (Minimum Variance Distortionless Response) beamformer.

[0072] Physical concept: "Beamforming" is similar to forming a "sound flashlight" in space, illuminating (receiving) sound only in the direction of the target while suppressing interference from other directions. Traditional Delayed Summation Beamforming (DSB) simply aligns the phases, which, while distortion-free, has weak interference suppression capabilities. MVDR minimizes output power (i.e., minimizes interference and noise from other directions) while maintaining distortion-free output in the target direction.

[0073] Mathematical derivation and formulas: Let the beamforming weight be The output is y= Its constraint is: regarding the target direction ,Require Signal attenuation is achieved through optimization objective (minimum variance): minimizing output power. ,in This is the covariance matrix (containing noise and interference). The optimal weights for MVDR are obtained using the Lagrange multiplier method. in, It is the MVDR beamforming weight vector (4x1 complex vector). It is the spatial covariance matrix The inverse matrix (4x4 matrix). It is the direction of the s-th voice part. The guiding vector (4x1 vector). The azimuth and pitch angles of the s-th voice part (unit: radians or degrees, real numbers). It is the conjugate transpose of the guiding vector (a 1x4 row vector). This is equivalent to a "whitening" filter, which assigns a small weight to the numerator in directions with high noise (high covariance). The steering vector is "distorted" according to the noise distribution, resulting in a null in the interference direction. Denominator normalization ensures that the gain in the target direction is always 1, allowing the MVDR beamformer to maintain unity gain in the target direction while automatically forming a null in the interference direction, improving the signal-to-interference ratio (SIR) by more than 20dB.

[0074] Spatial filter output: in, It is the first beam obtained after MVDR beamforming. The initial separation of each voice part is represented by its time-frequency representation (scalar complex number). It is the conjugate transpose of the weight vector (a 1x4 row vector). This is the four-channel input time-frequency vector (4x1 column vector). The output serves as the input for subsequent harmonic preservation processing, denoted as... For the sake of brevity, S6.2 and S6.3 still use [symbol name missing]. This represents the result.

[0075] S602: Mask Estimation for Harmonic Constraints Based on MVDR spatial filtering, a harmonic preservation factor is introduced when constructing a soft time-frequency mask: in, The soft time-frequency mask value of the s-th voice part at the k-th frequency point in the n-th frame is a real number that satisfies the following conditions: ; The time-frequency representation of the initial separation of the s-th voice part is a complex number; Indicates the first Time-frequency representation of competing voice parts Pick and , is a complex number; Indicates the index of the competing voice part, which is a positive integer; The absolute value squared represents the first The power spectral density values ​​of each voice part are non-negative real numbers; for From 1 to Summation means summation over all voices Summation operations, including equal ; This represents the estimated variance of the background noise, and is a non-negative real number. This represents the harmonic preservation factor, which is a scalar and a real number used to enhance the weighting of harmonic frequencies.

[0076] Harmonic retention factor Frequency-dependent Does it belong to the harmonic set? Harmonic set Criteria for judgment: 1. Fundamental frequency acquisition: The fundamental frequency estimate of the current frame of the voice part is obtained using steps S5.1 and S5.3. (Unit: Hz).

[0077] 2. Theoretical harmonic frequencies: For To consider the highest harmonic order (usually taken as 8~15), calculate the... The theoretical frequency of the subharmonic: 3. Mapping to discrete frequency points: Assume the frequency resolution of the STFT. The index of the center frequency corresponding to the theoretical harmonic is: in This is for rounding to the nearest integer.

[0078] 4. Frequency Range (Physical Bandwidth): Since actual harmonic energy will spread within a certain bandwidth near the theoretical frequency (affected by window function, frequency modulation, etc.), a frequency range of [missing information] is used. The neighborhood of the center is taken as the set of frequencies to which this harmonic belongs: The harmonic bandwidth half-width (unit: frequency index) corresponds to the physical frequency half-width. .

[0079] 5. Energy-assisted verification: To eliminate spurious harmonic frequencies caused by noise, further frequency verification can be performed. Satisfying the local peak condition: in For the reference microphone (or the initial separated signal) time-frequency amplitude, ≥1.0 (typical value 1.2). This condition can eliminate flat noise areas.

[0080] 6. All harmonic orders of Hejing: Noise / clean frequency points refer to regions that do not belong to any harmonic structure and whose energy is significantly lower than that of harmonics. They can be determined based on the following criteria: The non-harmonic condition is The low-energy condition is when the power spectral density at that frequency point is lower than a certain percentage of the average power across the entire frequency band of that frame (e.g., Or it can be defined directly as: for the unvoiced (non-periodic) part, all frequencies outside the harmonic set can be considered as noise / unvoiced frequency points. In practical implementation, this can be simplified: for each voice part, It can be taken as the set of all frequency points excluding harmonics that are below a certain absolute threshold (such as -40dB relative to maximum power).

[0081] Definition of harmonic preservation factor: in, Harmonic preservation factor (real number, >0). Harmonic enhancement factor ( ). The conflict zone suppression coefficient ( 1). For the first The first voice part in the n The set of harmonic frequency indexes of a frame (a set of integers containing the frequencies corresponding to all harmonics of that sound part). . For the first The first voice part in the n A set of noise / clear audio point indices for a frame (a set of integers).

[0082] For frequencies identified as harmonics, the mask value is increased (enhancement); for frequencies identified as noise, the normal value is maintained; for conflicting regions where sound parts overlap, the mask value is decreased (suppression). This "flexible processing" ensures that harmonics are not weakened while avoiding excessive mixing between sound parts.

[0083] S603: Harmonic continuity enhancement (frequency domain smoothing) Masking can cause tiny fluctuations in harmonic amplitude (sounding like a "trickle" or "drop-off"). These dips caused by masking are filled in by smoothing interpolation along the harmonic frequency direction. in, For the enhancement of harmonic continuity, the first The first voice part in the The time-frequency value (complex number) at the second harmonic. It is the first The fundamental frequency of each voice part (unit: Hz). The target harmonic order (positive integer, ). The summation variable (representing the harmonic index participating in smoothing, an integer) )·. The half-width of the harmonic smoothing window (a positive integer, usually 1-2). For the first The interpolation weights of the subharmonics (as...) Increases and decreases, usually with Gaussian weights or triangular weights, and non-negative real numbers. It is the sth voice part in the th... The original time-frequency value (complex number) at the subharmonic. To from arrive The summation operation. It is the normalization factor for the weights.

[0084] Effect: Maintains continuous variation in harmonic amplitude, resulting in a smoother and more natural sound. For non-harmonic frequencies, the original masked values ​​are preserved. final, This is the enhanced time-frequency representation output in step S6, which is used for time-domain reconstruction in step S7.

[0085] from In the end Complete data link: 1. Input: Multi-channel time-frequency representation 2. S6.1MVDR Spatial Filtering: 3.S6.2 Harmonic Constraint Mask Estimation: Utilizing Calculate the mask ,get 4. Harmonic Continuity Enhancement: Frequency domain smoothing is performed near the harmonic frequency, resulting in improved output. .

[0086] 5. Output: Enhanced single-channel time-frequency representation , used in S7.

[0087] S603: Spatial Filtering Refinement (MVDR Beamforming) Using the sound source direction information obtained in step S3, spatial filtering is performed using an MVDR (Minimum Variance Distortionless Response) beamformer. "Beamforming" is analogous to creating a "sound flashlight" in space, illuminating (receiving) only the sound from the target direction while suppressing interference from other directions. Traditional Delayed Summation Beamforming (DSB) simply aligns the phase; although distortion-free, its interference suppression capability is weak. MVDR minimizes output power (i.e., minimizes interference and noise from other directions) while maintaining distortion-free sound in the target direction.

[0088] S7: Time Domain Reconstruction and Harmonic Phase Synchronization After frequency domain separation, the signal needs to be converted back to the time domain (the sound we hear). This process requires special attention to phase consistency, as the human ear is sensitive to phase changes; discontinuous phase can lead to timbre distortion or a "metallic" sound. In particular, a fixed phase relationship (phase-locked loop) should be maintained between the harmonics of the human voice.

[0089] S701: Inverse Short-Time Fourier Transform (ISTFT) The processed time-frequency representation is converted into a time-domain waveform through inverse transform, and then the overlap-add method is used. in, For the ISTFT reconstructed number Each voice part is a time-domain signal (real number). It is a time variable (unit: seconds, continuous or discrete sample index). For frame index (sum across all time frames, ). The STFT frame length (positive integer, number of samples). Frequency index (summing and traversing from 0 to...) N-1 ). It is the first The first voice part in the Frame number Complex time-frequency value of a frequency point. It is a basis function (complex exponent) of the inverse discrete Fourier transform. It is a synthetic window function (time-domain representation, usually the same as the Hanning window, a real number). It is the frame shift (positive integer, number of samples). It is the DFT normalization coefficient. Frame index The summation operation. Frequency index The summation operation.

[0090] S702: Harmonic Phase Synchronization The harmonics of human voice should maintain a fixed phase relationship (usually in-phase or phase-locked). This relationship may be disrupted during separation and reconstruction. Phase-locking conditions can be achieved by forcibly adjusting the phase of each harmonic component: in, It is the first The first voice part The subharmonic at time The phase (unit: radians, real number). The fundamental frequency (first harmonic) of the first voice part at time t The phase (unit: radians, real number). It is the harmonic order (positive integer). It is the first Phase deviation of the subharmonic relative to the fundamental frequency (ideally 0, but in practice a small constant, a real number, in radians).

[0091] Implementation methods include: 1. Extract the fundamental frequency phase ; 2. Calculate the ideal harmonic phase ; 3. Adjust the phase of each harmonic component to align it with the ideal phase; 4. Maintain phase continuity and avoid the "clicking" sound caused by phase jumps; This maintains the "cohesion" and "warmth" of the vocals, preventing the separated sound from sounding "scattered" or "weak." Adaptive gain control and soft limiting are applied to the output signal to prevent clipping distortion while maintaining a natural dynamic range.

[0092] To ensure separation quality, the separation effect is monitored in real time. When a quality degradation is detected, parameters are adaptively adjusted to form a closed-loop control. Quality indicators include Harmonic Distortion Rate (HDR) and Inter-Sound-source Interference Ratio (ISIR).

[0093] Harmonic distortion rate : in, It is the harmonic distortion rate of the s-th voice (dimensionless, non-negative real number, the smaller the value, the smaller the distortion). It is the extracted h-th harmonic amplitude (a non-negative real number). It is the ideal first The amplitude of the second harmonic (reference value, non-negative real number). It is the harmonic order (summing over all considered harmonics, ). It is the square of the absolute value of a real number. Harmonic order The summation operation.

[0094] Acoustic leakage suppression ratio: in, Indicates from the first The sound part leaked to the first Interference ratio of each voice part (dimensionless, non-negative real number, the smaller the better). Let be the output signal (real number) of the s-th voice part. It is the time-domain impulse response or reference signal (real number) of the s'th voice part. It is the convolution operator. It is the square of the modulus of a complex or real number. For the time variable (sum across all time samples), ). It is a summation operation on the time sample t. <-20 dB indicates good separation.

[0095] When the HDR or ISIR exceeds the threshold, automatically adjust the band boundary in step S4 or the harmonic enhancement coefficient in step S6 to optimize the separation effect.

[0096] Embodiment 2 The following is a specific implementation description for "Where do you come from, my friend" in "Missing" according to the steps S1 - S8 of this embodiment: S1: Spatial acquisition and time delay compensation of the four - microphone tetrahedral array Arrange the four microphones in a tetrahedral geometry, Located at the origin , , , , with a spacing Take 10 cm, and the four - channel original signals are to .

[0097] Use the GCC - PHAT algorithm to estimate the time delay between channels. For the moment of the four - part chorus of the word "you", calculate the cross - power spectrum Divided by the amplitude , and obtain the generalized cross - correlation function through the inverse Fourier transform. Determine the time delay at the peak. For the 160 - ms rapid glide of the word "from", update the time delay estimate every 10 frames (about 26 ms) to ensure time alignment in the fast - transition section.

[0098] Perform time delay compensation on each channel signal: , to align the four - channel signals in time.

[0099] S: Time - frequency transformation and extraction of spatial statistical characteristics Perform short - time Fourier transform on the compensated signal. The frame length Take 512 (10.7 ms), the frame shift Take 128 (2.67 ms), add a Hanning window, and obtain the time - frequency representation .

[0100] At each time - frequency point Construct the spatial covariance matrix: The matrix element .

[0101] S3: Sound source spatial positioning and initial band division Use the MUSIC algorithm for spatial spectrum estimation. Perform eigen - decomposition on the covariance matrix to separate the signal subspace and the noise subspace. At the moment of the word "you", scan the azimuth angle and pitch angle , calculate the steering vector , obtain the spatial spectrum , and detect the spectral peak at the azimuth of female soprano +8°.

[0102] Four spectral peaks are detected in the character "朋": female soprano +8°, female alto -5°, male soprano +20°, male bass -15°. Calculate the energy of each vocal part band , initialize the band boundaries: female soprano 380 - 500 Hz, female alto 300 - 380 Hz, male soprano 250 - 300 Hz, male bass 180 - 240 Hz.

[0103] S4: Adaptive Band Optimization and Dynamic Boundary Adjustment Extract the spectral centroid of each vocal part and the spectral diffusivity . For the extremely fast glissando (392 → 440 Hz) of 160 ms in the character "从" (0:56), establish the objective function for band optimization , and use the greedy algorithm to recalculate the overlap every 5 frames (13.3 ms) , dynamically adjust the band boundaries to make the band center move from 416 Hz to 440 Hz during the glissando process. For the four - part harmony dense area of the character "朋", when the spectral overlap degree O > 1.3 is detected, immediately adjust the boundary between female soprano and female alto from 350 Hz to 365 Hz, and construct an adaptive filter bank , ensuring clear separation of the four voices.

[0104] S5: Harmonic Structure Detection, Modeling and Cross - Frame Tracking Adopt harmonic analysis and detect the fundamental frequency. For the candidate fundamental frequency of 440 Hz in the character "你", calculate , accumulate from the 1st to the 8th harmonics, and determine 440 Hz as the fundamental frequency of female soprano. Establish an ideal harmonic template .

[0105] Use the Viterbi algorithm for cross - frame tracking. For the glissando of the character "从", the state vector , adopt time - domain smoothing take 0.85 to adapt to rapid frequency changes. Track the time - varying harmonic trajectory of the long note "朋友" to ensure continuous harmonic structure.

[0106] The frame shift H = 128 samples (about 2.67 ms at a sampling rate of 48 kHz). The theoretical frequency change per frame in the glissando segment is about , but there may be a larger jump in practice due to the fundamental frequency estimation error. Set , allowing for fast and smooth tracking. With the guidance of the updated spectral centroid every 5 frames (13.3 ms) in S4, the Viterbi algorithm can accurately track the glissando trajectory from 392 Hz to 440 Hz without delay or "step" effects.

[0107] S6: Harmonic Preservation-based Vocal Part Separation and Enhancement Construct a soft time-frequency mask and introduce a harmonic preservation factor: . At 440 Hz of the female soprano for the character "朋", enhance the harmonics; in the conflict area (such as the second harmonic of the male bass at 392 Hz approaching the fundamental frequency of the female soprano), suppress interference. Perform harmonic continuity enhancement: for the long trill (±8 Hz) of the character "友", smoothly interpolate in the harmonic frequency direction, , maintaining the naturalness of the trill. Adopt MVDR beamforming for refinement: calculate the weights , maintain the gain in the +8° female soprano direction, form a zero in the -5° female alto direction, and output the spatial filtering .

[0108] S7: Time-domain Reconstruction and Harmonic Phase Synchronization Reconstruct the time-domain signal using the inverse short-time Fourier transform (ISTFT): , and use the overlap-add method to convert the processed time-frequency representation of 8 - 9 seconds into four-channel separated audio.

[0109] Perform harmonic phase synchronization: forcefully adjust the phases of the harmonic components of the four vocal parts of the character "朋" to meet the phase-locked condition , maintaining the cohesion of the human voice. Adopt relative phase preservation for the glissando of the character "从" to avoid clicks caused by phase jumps.

[0110] Calculate the harmonic distortion rate: , for the character "朋", detect the female soprano <0.1, determine high fidelity.

[0111] Calculate the vocal part leakage suppression ratio: , detect the leakage of the male bass in the female soprano channel < -18 dB, determine good separation. If or exceeds the threshold, automatically feedback and adjust the S4 frequency band boundary or the S6 harmonic enhancement coefficient.

[0112] Implemented through steps S1 - S8, four independent outputs are obtained: Female soprano track: The starting note of the character "你" is clean, the glissando of the character "从" is coherent, and 440 Hz of the character "朋" is prominent; Female alto track: The third-degree filling of 349 Hz of the character "朋" is clear; Male tenor track: The fifth-degree support of 262 Hz of the character "朋" is full; Male bass track: The root note of 196 Hz of the character "朋" is stable, without leakage of the female soprano.

[0113] Example 3 This embodiment describes in detail the function of each module of the system, such as Figure 2 As shown, the specific modules are as follows: Acquisition and compensation module 1 is configured to acquire multi-channel chorus signals, calculate the generalized cross-correlation function between pairs of microphone signals to estimate the inter-channel time delay, and perform time shift compensation on the multi-channel chorus signals based on the estimated inter-channel time delay to obtain aligned multi-channel signals; Transformation and statistics module 2 is configured to perform short-time Fourier transform on the aligned multi-channel signal to obtain the multi-channel time-frequency representation, and use the multi-channel time-frequency representation to calculate the spatial covariance matrix; Spatial positioning module 3 is configured to perform eigenvalue decomposition on the spatial covariance matrix to obtain the signal subspace and noise subspace, calculate the multi-signal classification spatial spectrum to determine the direction of each sound part, calculate the frequency band energy of each sound part, and initialize the frequency band boundary of each sound part. The frequency band optimization module 4 is configured to extract the spectral centroid and spectral spread of each sound part in the multi-channel time-frequency representation, adjust the frequency band boundary by maximizing the objective function that includes the signal-to-noise ratio, the inverse of the spectral overlap and the harmonic integrity measure, and construct a filter bank based on the adjusted frequency band boundary. Harmonic tracking module 5 is configured to use harmonic analysis and function to detect the fundamental frequency of each sound part, establish an ideal harmonic template using the detected fundamental frequency, and use the Viterbi algorithm to track the fundamental frequency and harmonic amplitude across frames to obtain the time-varying harmonic trajectory. The separation and enhancement module 6 is configured to use the guide vectors of each sound direction to construct a beamformer to perform spatial filtering on the multi-channel time-frequency representation to obtain a preliminary separated time-frequency representation, construct a soft time-frequency mask based on the time-varying harmonic trajectory and introduce a harmonic preservation factor, apply the soft time-frequency mask to the preliminary separated time-frequency representation and smooth the enhanced time-frequency representation in the harmonic frequency direction; The reconstruction synchronization module 7 is configured to perform an inverse short-time Fourier transform on the enhanced time-frequency representation to convert it into a separated time-domain signal, extract the fundamental frequency phase of the separated time-domain signal, and adjust the phases of each harmonic component to align with the ideal harmonic phase to obtain the target separated sound part signal.

[0114] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for adaptive frequency band optimization and choral vocal separation based on harmonic preservation, characterized in that, Includes the following steps: Multi-channel chorus signals are acquired, and the generalized cross-correlation function between each pair of microphone signals is calculated to estimate the inter-channel time delay. Based on the estimated inter-channel time delay, time shift compensation is performed on the multi-channel chorus signals to obtain aligned multi-channel signals. Short-time Fourier transform is performed on the aligned multi-channel signal to obtain the multi-channel time-frequency representation, and the spatial covariance matrix is ​​calculated using the multi-channel time-frequency representation; Eigenvalue decomposition of spatial covariance matrix is ​​performed to obtain signal subspace and noise subspace. Multi-signal classification spatial spectrum is calculated to determine the direction of each sound part. Frequency band energy of each sound part is calculated and frequency band boundaries of each sound part are initialized. The spectral centroid and spectral spread of each voice part in the multi-channel time-frequency representation are extracted. A greedy optimization strategy based on boundary movement is adopted. The frequency band boundary is adjusted by maximizing the objective function that considers the signal-to-noise ratio, the inverse of the spectral overlap, and the harmonic integrity metric. A filter bank is constructed based on the adjusted frequency band boundary to selectively filter the frequency band of each voice part and obtain the time-frequency representation after frequency band filtering. Harmonic analysis and function detection are used to detect the fundamental frequency of each voice part. An ideal harmonic template is established using the detected fundamental frequency to mark the theoretical distribution position of the harmonic components of each voice part. The Viterbi algorithm is used to track the fundamental frequency trajectory and amplitude changes of each harmonic across frames to obtain the time-varying harmonic trajectory. A beamformer is constructed using the guide vectors of each sound part to perform spatial filtering on the time-frequency representation after the frequency band filtering to obtain a preliminary separated time-frequency representation. A soft time-frequency mask with a harmonic preservation factor is constructed based on the time-varying harmonic trajectory. The soft time-frequency mask is applied to the preliminary separated time-frequency representation and the enhanced time-frequency representation is obtained by smoothing in the harmonic frequency direction. The enhanced time-frequency representation is converted into a separated time-domain signal by inverse short-time Fourier transform. The fundamental frequency phase of the separated time-domain signal is extracted and the phases of each harmonic component are adjusted to align with the ideal harmonic phase to obtain the target separated sound part signal.

2. The adaptive frequency band optimization choral vocal separation method based on harmonic preservation according to claim 1, characterized in that, The multi-channel chorus signal is acquired by microphones arranged at the four vertices of a three-dimensional coordinate system. The first microphone is located at the origin of the three-dimensional coordinate system, the second microphone is located at a set distance in the positive direction of the horizontal axis, the third microphone is located at a set distance in the positive direction of the vertical axis, and the fourth microphone is located at a set distance in the positive direction of the vertical axis.

3. The adaptive frequency band optimization choral vocal separation method based on harmonic preservation according to claim 1, characterized in that, The generalized cross-correlation function is calculated using the following formula: in, Indicates the first The microphone and the first The relative time delay between microphones, in seconds; This represents the integral variable of the time delay, i.e., the candidate time delay value, in seconds; Indicates the first Frequency domain representation of the microphone signal; Indicates the first Frequency domain conjugate representation of each microphone signal; The magnitude of the cross-power spectrum is represented by a non-negative real number. This represents the frequency integral variable, with the unit being Hz; Represents the imaginary unit; The power represents a complex exponential function, i.e., a phase rotation factor; This represents the time delay value that maximizes the integral. The operator.

4. The adaptive frequency band optimization choral vocal separation method based on harmonic preservation according to claim 1, characterized in that, The formula for calculating the objective function is as follows: in, This represents the objective function for frequency band optimization. Indicates the total number of voice parts; This represents the voice part index, with values ​​1, 2, ... ; Indicates the first Number of sub-bands per voice part; This represents the sub-band index, with values ​​1, 2, ... ; Indicates the first The first voice part in the The signal-to-noise ratio of each subband; Indicates the first The first voice part in the Spectral overlap between individual sub-bands and other sound parts; This represents the harmonic integrity measure of the s-th voice part in the m-th subband, satisfying 0 ≤ ≤1; , , Let represent the weighting coefficient, which is a non-negative real number, satisfying . It equals 1.

5. The adaptive frequency band optimization choral vocal separation method based on harmonic preservation according to claim 4, characterized in that, The formula for calculating the spectral overlap is: in, For the first The first voice part in the Spectral overlap of each subband For indexing competing voice parts, Index the start and end frequency points covered by this sub-band. ) is the first Each voice part at the time frequency point The complex time-frequency amplitude, For the time-frequency amplitude of the competing voice parts, To prevent overflow of constants.

6. The adaptive frequency band optimization choral vocal separation method based on harmonic preservation according to claim 1, characterized in that, The harmonic analysis and function are as follows: in, Harmonic analysis and functions at candidate fundamental frequencies The value at; It is a candidate fundamental frequency; It is the harmonic order; It is the highest harmonic order considered; The mixed signal is in the first... Frame, frequency Time-frequency representation at a location; It is the amplitude of that frequency component; It is the first Attenuation weight of subharmonics.

7. The adaptive frequency band optimization choral vocal separation method based on harmonic preservation according to claim 1, characterized in that, The harmonic preservation factor in the soft time-frequency mask is assigned values ​​according to the set to which the frequency point belongs, as follows: in, Harmonic preservation factor For harmonic enhancement coefficient, This represents the conflict zone suppression coefficient. For the first The first voice part in the n The set of harmonic frequency indexes of a frame. For the first The first voice part in the n A set of indexes for noise / clear audio points in a frame.

8. The adaptive frequency band optimization choral vocal separation method based on harmonic preservation according to claim 1, characterized in that, The smoothing in the harmonic frequency direction involves calculating the sum of the products of the interpolation weights of each harmonic and the original time-frequency value within the upper and lower limits of the target harmonic order, and then dividing the sum of the products by the sum of the interpolation weights to obtain the enhanced time-frequency value.

9. The adaptive frequency band optimization choral vocal separation method based on harmonic preservation according to claim 1, characterized in that, After acquiring the target separated acoustic signal, the harmonic distortion rate and the acoustic leakage suppression ratio are calculated. The harmonic distortion rate is calculated based on the square of the absolute value of the difference between the extracted harmonic amplitude and the ideal harmonic amplitude. The acoustic leakage suppression ratio is calculated based on the square of the absolute value of the convolution between the target separated acoustic signal and the competing acoustic reference signal. If either the harmonic distortion rate or the acoustic leakage suppression ratio is greater than a preset threshold, at least one of the harmonic enhancement coefficients in the frequency band boundary and the soft time-frequency mask is readjusted until the recalculated harmonic distortion rate is not greater than 0.1 and the recalculated acoustic leakage suppression ratio is not greater than -18dB.

10. An adaptive frequency band optimization choral vocal separation system based on harmonic preservation, characterized in that, Include: The acquisition and compensation module is configured to acquire multi-channel chorus signals, calculate the generalized cross-correlation function between each pair of microphone signals to estimate the inter-channel delay, and perform time-shift compensation on the multi-channel chorus signals based on the estimated inter-channel delay to obtain aligned multi-channel signals. The transformation and statistics module is configured to perform short-time Fourier transform on the aligned multi-channel signal to obtain the multi-channel time-frequency representation, and use the multi-channel time-frequency representation to calculate the spatial covariance matrix; The spatial positioning module is configured to perform eigenvalue decomposition on the spatial covariance matrix to obtain the signal subspace and noise subspace, calculate the multi-signal classification spatial spectrum to determine the direction of each sound part, calculate the frequency band energy of each sound part, and initialize the frequency band boundary of each sound part. The frequency band optimization module is configured to extract the spectral centroid and spectral spread of each sound part in the multi-channel time-frequency representation, adjust the frequency band boundary by maximizing the objective function that includes the signal-to-noise ratio, the inverse of the spectral overlap, and the harmonic integrity metric, and construct a filter bank based on the adjusted frequency band boundary. The harmonic tracking module is configured to use harmonic analysis and functions to detect the fundamental frequency of each sound part, establish an ideal harmonic template using the detected fundamental frequency, and use the Viterbi algorithm to track the fundamental frequency and harmonic amplitude across frames to obtain the time-varying harmonic trajectory. The separation and enhancement module is configured to use the guide vectors of each sound part to construct a beamformer to perform spatial filtering on the multi-channel time-frequency representation to obtain a preliminary separated time-frequency representation. Based on the time-varying harmonic trajectory, a soft time-frequency mask with a harmonic preservation factor is constructed. The soft time-frequency mask is applied to the preliminary separated time-frequency representation and the enhanced time-frequency representation is obtained by smoothing in the harmonic frequency direction. The reconstruction synchronization module is configured to perform inverse short-time Fourier transform on the enhanced time-frequency representation to convert it into a separated time-domain signal, extract the fundamental frequency phase of the separated time-domain signal, and adjust the phases of each harmonic component to align with the ideal harmonic phase to obtain the target separated sound part signal.

Images