A karaoke audio processing method based on human voice separation and repair

Abstract

The application discloses a karaoke audio processing method based on human voice separation and repair, which comprises the following steps: acquiring a time-frequency matrix of a karaoke audio signal, and judging whether noise components exist in the karaoke audio signal; identifying a noise basis matrix and an effective audio basis matrix, calculating the overlap coefficients of noise and effective audio on different frequency bands, and determining the cross frequency bands; identifying transient noise and steady noise according to the noise basis matrix and the corresponding activation matrix; designing a filter to perform real-time filtering on noise of each frame; judging the lost phase information, using linear difference interpolation to fill the missing area phase, and compensating the time delay through time-frequency matrix translation; judging the denoising effect and the audio repair effect, and formulating and executing a karaoke audio signal processing improvement scheme. The application can effectively solve the problems of noise interference, audio distortion, phase loss and time delay compensation existing in the current karaoke application, significantly improve the audio quality, and provide users with a more high-quality and personalized karaoke experience.

Description

Technical Field

[0001] This invention relates to the field of next-generation information technology, and in particular to a karaoke audio processing method based on human voice separation and restoration. Background Technology

[0002] With the increasing popularity of karaoke entertainment apps, users' demands for audio quality are growing, especially in terms of sound quality, clarity, audio latency, and the synchronization of vocals and accompaniment. However, most current karaoke apps still face problems such as background noise, audio signal distortion, and poor synchronization between vocals and accompaniment, preventing users from enjoying ideal audio effects during karaoke. Traditional audio denoising techniques are usually based on simple frequency domain analysis and filtering methods, such as filtering audio signals by setting a fixed energy threshold. This method is prone to audio distortion in practical applications, especially when there is strong overlap between the vocal and accompaniment frequency bands, making it difficult to guarantee denoising effectiveness. In the audio denoising process, handling the overlapping frequency bands of noise and effective audio, especially transient and steady-state noise, remains a challenge. Furthermore, existing audio restoration techniques often focus on single problems, such as phase recovery or delay compensation, but few comprehensive solutions can simultaneously address noise suppression, audio restoration, and the synchronization of vocals and accompaniment. The loss of phase information and delay compensation issues in existing technologies often result in a lack of naturalness in the repaired audio signal, especially during multi-person karaoke sessions, where the asynchrony between the accompaniment and vocals severely impacts the user experience. Current solutions for audio denoising, restoration, and enhancement are inadequate and cannot meet the demands of karaoke users for high-quality audio. Therefore, a multi-dimensional karaoke audio processing method is urgently needed to improve audio signal quality. Summary of the Invention

[0003] This invention addresses the problems existing in the prior art by providing a karaoke audio processing method based on human voice separation and restoration, mainly comprising:

[0004] The audio acquisition module acquires the karaoke audio signal, and the short-time Fourier transform algorithm is used to transform the karaoke audio signal from the time domain to the frequency domain to obtain the time-frequency matrix of the karaoke audio signal, and to determine whether there are noise components in the karaoke audio signal.

[0005] Based on the time-frequency matrix of the karaoke audio signal, nonnegative matrix decomposition is used to obtain the basis matrix and activation matrix, identify the noise basis matrix and effective audio basis matrix, calculate the overlap coefficient of noise and effective audio in different frequency bands, and determine the crossover frequency band.

[0006] Based on the noise basis matrix and the corresponding activation matrix, the energy value, zero-crossing rate, spectral flatness and statistical characteristics of each frame of the noise signal are determined, and transient noise and steady-state noise are identified.

[0007] Based on the temporal envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal, an adaptive filtering algorithm is used to design a filter to filter the noise of each frame in real time, thereby obtaining the denoised karaoke audio signal.

[0008] By comparing the amplitude spectrum and phase spectrum of the denoised karaoke audio signal, the lost phase information is determined. The envelope is calculated in combination with the amplitude spectrum, the phase of the missing region is interpolated using linear interpolation, and the time delay is compensated by shifting the time-frequency matrix.

[0009] Based on user feedback data, we assess the noise reduction and audio restoration effects of karaoke audio signals, and formulate and implement improvement plans for karaoke audio signal processing.

[0010] Furthermore, the step of acquiring the karaoke audio signal through the audio acquisition module, converting the karaoke audio signal from the time domain to the frequency domain using the short-time Fourier transform algorithm, obtaining the time-frequency matrix of the karaoke audio signal, and determining whether there are noise components in the karaoke audio signal includes:

[0011] The audio acquisition module acquires the karaoke audio signal. Based on a preset frame length and frame shift, the audio signal is divided into several overlapping short-time frame sequences. For each frame, a Hamming window is applied, and a short-time Fourier transform algorithm is used to transform the karaoke audio signal from the time domain to the frequency domain. The spectrum of each frame is calculated, and the time-frequency matrix of the karaoke audio signal, including the amplitude spectrum and phase spectrum, is obtained. Based on the amplitude spectrum of the karaoke audio signal, the power spectrum and spectral flatness of each frame are calculated to determine the energy distribution of each frequency band. Based on the energy distribution and spectral flatness of each frequency band, a support vector machine algorithm is used to train the model and construct a noise component discrimination model to determine whether there are noise components in the karaoke audio signal.

[0012] Further, the process of obtaining the basis matrix and activation matrix using nonnegative matrix decomposition based on the time-frequency matrix of the karaoke audio signal, identifying the noise basis matrix and the effective audio basis matrix, calculating the overlap coefficient of noise and effective audio in different frequency bands, and determining the crossover frequency band includes:

[0013] Based on the time-frequency matrix of the karaoke audio signal, a nonnegative matrix decomposition of the amplitude spectrum is used. The number of basis functions and iterations are set, and KL divergence is used as the loss function to decompose the amplitude spectrum of the karaoke audio signal. The basis matrix and activation matrix are alternately optimized using a multiplication update rule. The basis matrix represents the fundamental frequency components of the audio signal, and the activation matrix represents the activation level of the fundamental frequency components in each frame. Based on the column vectors of the basis matrix, a K-maens clustering algorithm is used to perform cluster analysis to identify the noise basis matrix and the effective audio basis matrix. The effective audio includes vocals and accompaniment. Based on the activation matrices corresponding to the noise basis matrix and the effective audio basis matrix, a cosine similarity calculation method is used to calculate the similarity between the activation values ​​of the noise component and the activation values ​​of the effective audio component in each frequency band, determining the overlap coefficient between noise and effective audio in different frequency bands. If the overlap coefficient is greater than a preset threshold, the frequency band is determined to be an intersecting frequency band.

[0014] Further, the step of determining the energy value, zero-crossing rate, spectral flatness, and statistical characteristics of each frame of the noise signal based on the noise basis matrix and the corresponding activation matrix, and identifying transient noise and steady-state noise, includes:

[0015] Based on the noise basis matrix and the corresponding activation matrix, the short-time energy calculation formula is used. The energy value of each frame of the noise signal is obtained to determine the temporal envelope characteristics of the noise signal. Here, E(t) is the audio signal energy at time t, X(t,f) is the amplitude spectrum value at time t and frequency f, and F is the frequency range. The temporal envelope characteristics represent the energy fluctuation amplitude of the noise signal. Based on the noise basis matrix, the zero-crossing rate and spectral flatness of each frame of the noise signal are calculated. The zero-crossing rate is the number of times the signal crosses zero points per unit time. The energy value difference of each frame of the noise signal is calculated. If there is a short-time frame sequence with an energy value difference greater than a preset difference threshold or a zero-crossing rate greater than a preset zero-crossing rate threshold, the noise in that frame is marked as transient noise; otherwise, it is marked as steady-state noise. By comparing the energy value of each frame with the energy values ​​of the preceding and following frames, the duration and intensity of transient noise are determined. Statistical methods are used to calculate the mean, variance, kurtosis, and skewness of the noise signal energy to determine the statistical characteristics of the noise signal.

[0016] Furthermore, based on the time-domain envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal, an adaptive filtering algorithm is used to design a filter to filter the noise in each frame in real time, resulting in a denoised karaoke audio signal, including:

[0017] Based on the temporal envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal, an adaptive filtering algorithm is used to design a filter and determine the initial parameters of the filter. Real-time filtering is performed on the noise of each frame. The initial parameters of the filter include, but are not limited to, the filter length, bandwidth, and gain coefficient. If a short frame of transient noise is identified in the noise signal, an energy adjustment factor is calculated based on the energy of the noise signal and the effective audio in that frame, and the denoising intensity of each frame's audio signal is adjusted based on the energy adjustment factor. If the energy adjustment factor is greater than a preset threshold, the gain coefficient of the filter is increased or the bandwidth is reduced based on the energy adjustment factor. The filter parameters are dynamically adjusted using the temporal envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal in the real-time acquired karaoke audio signal. The filter removes the noise components from the original karaoke audio signal, resulting in a denoised karaoke audio signal, and a time-frequency matrix of the denoised karaoke audio signal is generated.

[0018] It also includes, if a short frame of transient noise is identified, calculating an energy adjustment factor based on the noise signal and the energy of the effective audio in that frame, and adjusting the denoising intensity of each frame's audio signal based on the energy adjustment factor, specifically including:

[0019] If a short frame of transient noise is identified, the energy difference ratio between the noise source and the valid audio source is calculated based on the energy of the noise signal and the valid audio signal in that frame. Among them, E n (t) is the energy of the noise signal, E a (t) is the energy of the effective audio source; based on the energy difference ratio between the noise source and the effective audio source, the energy adjustment factor formula is used. Calculate the energy regulation factor α(t) ) , where β and γ are adjustment parameters used to control the sensitivity of noise and audio balance, obtained by fitting historical data; the denoising intensity of each frame of audio signal is dynamically adjusted according to the energy adjustment factor.

[0020] Furthermore, the step of determining the lost phase information by comparing the amplitude spectrum and phase spectrum of the denoised karaoke audio signal, calculating the envelope by combining the amplitude spectrum, using linear interpolation to interpolate the phase of the missing region, and compensating for the time delay by shifting the time-frequency matrix includes:

[0021] By comparing the amplitude and phase spectra of the denoised karaoke audio signal, it is determined whether there is any lost phase information. If phase information is lost, the frequency range of the lost phase is determined by examining the missing regions in the phase spectrum. At each frequency point, the instantaneous phase information of the signal is obtained by combining the logarithm of the amplitude spectrum with the phase spectrum, and the envelope of the signal is calculated based on the amplitude spectrum. Based on the known instantaneous phase and envelope information of adjacent frequency points, a linear interpolation method is used to interpolate the missing phase region to recover the lost phase information. Phase information, combined with the amplitude spectrum of the denoised karaoke audio signal, is used to reconstruct the time-frequency matrix of the repaired karaoke audio signal. Based on the time-frequency matrix of the repaired karaoke audio signal, a dynamic time warping algorithm is used to match the time axis of the audio signal and calculate the time delay between the input and output of the karaoke audio signal. Based on the time delay calculation result, the time column in the time-frequency matrix of the repaired karaoke audio signal is shifted to obtain the time-frequency matrix of the delay-compensated karaoke audio signal. The inverse short-time Fourier transform is used to convert the time-frequency matrix back to the time domain signal to obtain the output signal of the karaoke audio signal.

[0022] Furthermore, the step of judging the noise reduction and audio restoration effects of the karaoke audio signal based on user feedback data, and formulating and implementing an improved karaoke audio signal processing plan, includes:

[0023] By setting up a feedback mechanism, user feedback data on sound quality, audio latency, vocal clarity, and accompaniment synchronization after the user's karaoke experience is obtained to determine the noise reduction and audio restoration effects of the karaoke audio signal. If the noise reduction or audio restoration effect of the karaoke audio signal does not meet the preset effect standard, an improvement plan for karaoke audio signal processing is formulated and implemented based on the user feedback data until the noise reduction and audio restoration effects of the karaoke audio signal both meet the preset effect standard. The improvement plan includes, but is not limited to, adjusting filter parameters, optimizing phase interpolation methods to fill in lost phase information, and readjusting latency compensation through dynamic time warping.

[0024] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0025] This invention provides a karaoke audio processing method based on vocal separation and restoration. This invention accurately identifies noise and effective audio components, especially in complex situations where noise and audio signal frequencies overlap. It effectively distinguishes the frequency bands of noise and effective audio components, particularly addressing the complex problem of overlapping noise and audio components. By calculating the overlap coefficient, it determines the cross-frequency band, accurately removing noise while preserving the core components of the audio signal. This invention provides the ability to dynamically adjust the denoising intensity to differentiate between transient and steady-state noise, optimizing the denoising effect in a timely manner based on noise changes. By comparing the amplitude spectrum and phase spectrum, this invention restores lost phase information, ensuring the naturalness and synchronization of the audio signal. By incorporating user feedback for real-time optimization, dynamically adjusting filter parameters and restoration strategies, it can improve the audio processing effect in real time according to user needs, providing personalized audio restoration solutions. This invention provides a karaoke audio processing method based on vocal separation and restoration, effectively solving problems such as noise interference, audio distortion, phase loss, and delay compensation in current karaoke applications, significantly improving audio quality and providing users with a higher quality and more personalized karaoke experience. Attached Figure Description

[0026] Figure 1 This is a flowchart of a karaoke audio processing method based on human voice separation and restoration according to the present invention;

[0027] Figure 2 This is a schematic diagram of a karaoke audio processing method based on human voice separation and restoration according to the present invention;

[0028] Figure 3 This is another schematic diagram of a karaoke audio processing method based on human voice separation and restoration according to the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] like Figure 1-3 This embodiment of a karaoke audio processing method based on human voice separation and restoration may specifically include:

[0031] Step S101: Acquire the karaoke audio signal through the audio acquisition module, use the short-time Fourier transform algorithm to convert the karaoke audio signal from the time domain to the frequency domain, obtain the time-frequency matrix of the karaoke audio signal, and determine whether there are noise components in the karaoke audio signal.

[0032] The audio acquisition module acquires the karaoke audio signal and divides it into several overlapping short-time frame sequences based on preset frame length and frame shift. For each frame, a Hamming window is applied, and a short-time Fourier transform algorithm is used to transform the karaoke audio signal from the time domain to the frequency domain. The spectrum of each frame is calculated, obtaining the time-frequency matrix of the karaoke audio signal, including the amplitude spectrum and phase spectrum. Based on the amplitude spectrum of the karaoke audio signal, the power spectrum and spectral flatness of each frame are calculated to determine the energy distribution of each frequency band. Based on the energy distribution and spectral flatness of each frequency band, a support vector machine algorithm is used to train a model to construct a noise component discrimination model, determining whether noise components exist in the karaoke audio signal.

[0033] For example, a karaoke audio signal is being processed. This signal contains vocals and background noise. The audio signal is acquired through an audio acquisition module with a sampling rate of 16kHz and a duration of 5 seconds, meaning the acquired audio signal has a total of 80,000 sampling points. Based on a preset frame length and frame shift, the entire audio signal is divided into multiple overlapping short-time frame sequences. If the frame length is set to 512 sampling points and the frame shift to 256 sampling points, each frame contains 512 sampling points, and there is a 256-sampling-point overlap between frames, resulting in approximately 320 frames. Each frame's signal is processed using a Hamming window function to smooth transitions at boundaries, thereby reducing spectral leakage. A short-time Fourier transform is used to convert the time-domain signal of each frame into a frequency-domain signal. The spectrum of a given frame might show that the low-frequency portion is mainly composed of frequency components at 100Hz, 200Hz, and 300Hz, while the high-frequency portion might mainly consist of frequency components at 1kHz and 2kHz. After obtaining the spectrum of each frame, the power spectrum of each frame is calculated by squaring the amplitude of the spectrum. If, for a given frame, the calculated amplitude spectrum shows that the frequency components have an amplitude of 2.0 at 50Hz, 1.5 at 200Hz, 3.0 at 1kHz, and 0.5 at 2kHz, then each frequency component in the power spectrum is the square of these amplitude values, which are 4.0, 2.25, 9.0, and 0.25 respectively. These power spectrum values ​​represent the energy distribution of each frequency component, that is, the energy contribution of the frame at different frequencies. If the calculated spectral flatness of 50Hz, 200Hz, 1kHz, and 2kHz is 0.4, 0.2, 0.5, and 0.8 respectively, and historical data on the energy distribution and spectral flatness of each frequency band are obtained, a support vector machine algorithm is used to train the model based on the energy distribution and spectral flatness of each frequency band, thus constructing a noise component discrimination model. Based on the trained noise component discrimination model, when a new audio signal is input, the noise component discrimination model will determine in real time whether there are noise components in the audio signal.

[0034] Step S102: Based on the time-frequency matrix of the karaoke audio signal, non-negative matrix decomposition is used to obtain the basis matrix and activation matrix, identify the noise basis matrix and effective audio basis matrix, calculate the overlap coefficient of noise and effective audio in different frequency bands, and determine the cross-band.

[0035] Based on the time-frequency matrix of the karaoke audio signal, a nonnegative matrix decomposition of the amplitude spectrum is employed. The number of basis functions and iterations are set, and KL divergence is used as the loss function to decompose the amplitude spectrum of the karaoke audio signal. The basis matrix and activation matrix are alternately optimized using a multiplicative update rule. The basis matrix represents the fundamental frequency components of the audio signal, and the activation matrix represents the activation level of the fundamental frequency components in each frame. Based on the column vectors of the basis matrix, a K-maens clustering algorithm is used to perform cluster analysis, identifying the noise basis matrix and the effective audio basis matrix. The effective audio includes vocals and accompaniment. Based on the activation matrices corresponding to the noise basis matrix and the effective audio basis matrix, a cosine similarity calculation method is used to calculate the similarity between the activation values ​​of the noise component and the activation values ​​of the effective audio component in each frequency band, determining the overlap coefficient between noise and effective audio in different frequency bands. If the overlap coefficient is greater than a preset threshold, the frequency band is determined to be an interleaved frequency band.

[0036] For example, consider a 5-second karaoke audio signal with a sampling rate of 16kHz, resulting in 80,000 sampling points. Divide the audio signal into multiple overlapping short-time frame sequences, each frame consisting of 512 sampling points, with a frame shift of 256 sampling points, resulting in approximately 320 frames. Use a short-time Fourier transform to convert the audio signal from the time domain to the frequency domain, obtaining a time-frequency matrix containing the amplitude and phase spectra of the audio signal. Use non-negative matrix decomposition, setting the number of basis functions to 10 (meaning the frequency components of the audio signal will be decomposed into 10 basis functions), and setting the iteration count to 1000. Non-negative matrix decomposition yields two matrices: the basis matrix represents the fundamental frequency components of the audio signal, with each column representing a fundamental frequency component; the activation matrix represents the activation level of each fundamental frequency component in each frame (e.g., the first column of the basis matrix might represent a low-frequency component, the second a mid-frequency component, and the third a high-frequency component); and the activation matrix represents the intensity of each frequency component in each frame. If, after nonnegative matrix factorization, certain columns of the resulting basis matrix contain the following frequency components: the first column has a frequency of 100Hz and a fundamental frequency amplitude of 0.8; the second column has a frequency of 500Hz and a fundamental frequency amplitude of 0.5; the third column has a frequency of 1000Hz and a fundamental frequency amplitude of 0.3; and the fourth column has a frequency of 1500Hz and a fundamental frequency amplitude of 0.2, then K-means clustering is used to perform cluster analysis based on the column vectors of the basis matrix. The goal is to separate these fundamental frequency components into a noise basis matrix and an effective audio basis matrix. Using K=2 as the number of clusters, the K-means algorithm will divide these fundamental frequency components into two categories: one category contains the effective audio components with fundamental frequencies of 100Hz, 500Hz, and 10000Hz; the other category contains the noise audio components with a fundamental frequency of 1500Hz. In different columns of the activation matrices corresponding to the noise basis matrix and the effective audio basis matrix, the activation value of each frequency component changes over time. To measure the overlap between the noise and effective audio basis matrices in a specific frequency band, the activation value in each frequency band is selected, i.e., the row of the corresponding activation matrix. If the activation values ​​of the noise basis matrix in a certain time frame are [0.6, 0.7, 0.4, 0.3] and the activation values ​​of the effective audio basis matrix are [0.7, 0.6, 0.3, 0.4], and their cosine similarity is calculated to be 0.8, then the overlap coefficient between noise and effective audio in the 1000Hz frequency band is 0.8, which is greater than the preset coefficient threshold of 0.75. This means that there is a strong overlap between noise and effective audio components in this frequency band, therefore this frequency band is an intersecting frequency band.

[0037] Step S103: Based on the noise basis matrix and the corresponding activation matrix, determine the energy value, zero-crossing rate, spectral flatness and statistical characteristics of each frame of the noise signal, and identify transient noise and steady-state noise.

[0038] Based on the noise basis matrix and the corresponding activation matrix, the short-time energy calculation formula is used. The energy value of each frame of the noise signal is obtained to determine the temporal envelope characteristics of the noise signal. Here, E(t) is the audio signal energy at time t, X(t,f) is the amplitude spectrum value at time t and frequency f, F is the frequency range, and the temporal envelope characteristics represent the energy fluctuation amplitude of the noise signal. Based on the noise basis matrix, the zero-crossing rate and spectral flatness of each frame of the noise signal are calculated. The zero-crossing rate is the number of times the signal crosses zero points per unit time. The energy value difference of each frame of the noise signal is calculated. If there is a short-time frame sequence with an energy value difference greater than a preset difference threshold or a zero-crossing rate greater than a preset zero-crossing rate threshold, the noise in that frame is marked as transient noise; otherwise, it is marked as steady-state noise. By comparing the energy value of each frame with the energy values ​​of the preceding and following frames, the duration and intensity of transient noise are determined. Statistical methods are used to calculate the mean, variance, kurtosis, and skewness of the noise signal energy to determine the statistical characteristics of the noise signal.

[0039] For example, consider a karaoke audio signal with a sampling rate of 16kHz and a duration of 3 seconds. This means the audio signal has a total of 48,000 sampling points. The audio signal is divided into multiple overlapping frames using a short-time Fourier transform. Each frame has a length of 512 sampling points, and the frame shift is 256 sampling points, resulting in approximately 200 frames in the audio. The amplitude spectrum |X(t,f)| of each frame is calculated using the time-frequency matrix, and the short-time energy is calculated using the formula... Calculate the energy of this frame, where E(t) is the audio signal energy at time t, X(t,f) is the amplitude spectrum value at time t and frequency f, F is the frequency range, and the time-domain envelope characteristic is the energy fluctuation amplitude of the noise signal. If the amplitude spectrum of the first frame, i.e., at t=1, is |X(1,50)|=1.5, |X(1,200)|=0.8, |X(1,1000)|=2.0, |X(1,3000)|=0.5, then the total energy of the first frame is 7.14. In the second frame, at t=2, the amplitude spectrum is |X(2, 50)| = 2.0, |X(2, 200)| = 1.2, |X(2, 1000)| = 1.8, and |X(2, 3000)| = 0.6, so the total energy of the second frame is 8.04. Based on the noise basis matrix, the zero-crossing rate and spectral flatness of the noise signal are calculated. The zero-crossing rate is the number of times the signal crosses zero points per unit time. The zero-crossing rate of the first frame is 10 times, and the spectral flatness of the first frame is 0.872. The zero-crossing rate of the second frame is 25 times, and the spectral flatness of the second frame is 0.906. By observing the change in the zero-crossing rate, it can be determined whether the signal is transient noise. Since the zero-crossing rate of the second frame is higher, it indicates that the change is more drastic, so it can be marked as transient noise. By calculating the energy difference, i.e., the energy difference between two adjacent frames, the energy of the first frame is 7.14, while the energy of the second frame is 8.04, and the energy difference between the first and second frames is 0.90. If the preset energy difference threshold is 0.5, then this difference is greater than the threshold, so the second frame can be considered transient noise. Finally, statistical methods are used to calculate the mean, variance, kurtosis, and skewness of the noise signal, yielding a mean of 7.6, a variance of 0.92, a kurtosis of 4.1, and a skewness of 0.3.

[0040] Step S104: Based on the temporal envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal, an adaptive filtering algorithm is used to design a filter to filter the noise of each frame in real time, thereby obtaining the denoised karaoke audio signal.

[0041] Based on the temporal envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal, an adaptive filtering algorithm is used to design a filter and determine its initial parameters. Noise is filtered in real-time for each frame. These initial parameters include, but are not limited to, the filter length, bandwidth, and gain coefficient. If a short frame of transient noise is identified in the noise signal, an energy adjustment factor is calculated based on the energy of the noise signal and the effective audio in that frame. The denoising intensity of each frame's audio signal is adjusted based on this energy adjustment factor. If the energy adjustment factor is greater than a preset threshold, the filter's gain coefficient is increased or the bandwidth is reduced based on the energy adjustment factor. The filter parameters are dynamically adjusted using the temporal envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal in the real-time acquired karaoke audio signal. The filter removes the noise components from the original karaoke audio signal, resulting in a denoised karaoke audio signal, and a time-frequency matrix of the denoised karaoke audio signal is generated.

[0042] For example, during the processing, the amplitude spectrum and power spectrum of each frame have been calculated, and the time-domain envelope feature of the noise signal is obtained as 0.8, the spectral flatness is 0.3, and the statistical characteristics include a mean of 1.5, a variance of 0.2, a kurtosis of 3.5, and a skewness of 0.1. The frequency components in certain columns of the basis matrix include the first column with a frequency of 100Hz and a fundamental frequency component amplitude of 0.8, the second column with a frequency of 500Hz and a fundamental frequency component amplitude of 0.5, the third column with a frequency of 1000Hz and a fundamental frequency component amplitude of 0.3, the fourth column with a frequency of 1500Hz and a fundamental frequency component amplitude of 0.2, the crossover frequency band is 1000Hz, and the overlap coefficient is 0.8. Based on the above characteristics, a least mean square adaptive filter is designed using the least mean square algorithm to remove noise signals in real time. The initial parameters of the filter include: a filter length of 64 sampling points, meaning the filter processes each frame based on historical data from 64 sampling points; a bandwidth of 200Hz to adapt to the current noise frequency range; and an initial gain coefficient of 1.2, representing the filter's denoising strength. If a short frame of transient noise is identified in the noise signal, an energy adjustment factor is calculated based on the energy of the noise signal and the effective audio in that frame. The denoising strength of the filter is adjusted according to the energy ratio of the noise to the effective audio. If the noise energy of the current frame is 2.0 and the effective audio energy is 6.0, the energy adjustment factor α(t)≈0.474 can be calculated using the formula α(t)≈0.474. Based on the calculated energy adjustment factor α(t), the filter's gain coefficient and bandwidth can be dynamically adjusted. Since α(t) ≈ 0.474, which is less than the preset factor threshold of 0.7, the filter's gain coefficient remains at 1.2, requiring no further enhancement of noise reduction. The 200Hz bandwidth is suitable for the current noise frequency range, so the filter's bandwidth remains unchanged. The time-domain envelope characteristics, spectral flatness, statistical properties, basis matrix, crossbands, and overlap coefficients of the noise signal in the real-time acquired karaoke audio signal are used to dynamically adjust the filter parameters. For example, as the audio signal changes, the zero-crossing rate of some frames may increase significantly, leading to an increase in the filter's gain coefficient. In this case, the filter's gain coefficient may increase to 1.5, and the bandwidth may be reduced to 150Hz to more accurately remove transient noise. An adaptive filter is used to filter the noise of each frame of the audio signal in real time. If the noise energy is high in the 10th frame, the filter's gain coefficient increases to 1.5, and the bandwidth is reduced. After filtering, significant background noise is successfully removed, while retaining effective audio components. Ultimately, the time-frequency matrix of the denoised audio signal showed a significant reduction in noise components in the amplitude and phase spectra, while the vocals and accompaniment were preserved.

[0043] If a short frame of transient noise is identified, an energy adjustment factor is calculated based on the noise signal and the energy of the effective audio in that frame, and the denoising intensity of each frame's audio signal is adjusted based on the energy adjustment factor.

[0044] If a short frame of transient noise is identified, the energy difference ratio between the noise source and the valid audio source is calculated based on the energy of the noise signal and the valid audio signal in that frame. Among them, E n (t) is the energy of the noise signal, E a (t) represents the energy of the effective audio source. The energy adjustment factor formula is used based on the energy difference ratio between the noise source and the effective audio source. The energy adjustment factor α(t) is calculated, where β and γ are adjustment parameters used to control the sensitivity of noise and audio balance, and are obtained by fitting historical data. Based on the energy adjustment factor, the denoising intensity of each frame of the audio signal is dynamically adjusted.

[0045] For example, the 50th frame of a karaoke audio signal contains transient noise with an energy of 3.2 and an effective audio energy of 8.0. By calculating the ratio of these two, the energy difference ratio is obtained. =0.4, based on the energy difference ratio, use the energy adjustment factor formula. The energy adjustment factor α(t) is calculated, where β and γ are adjustment parameters used to control the sensitivity of noise and audio balance, obtained by fitting historical data. If β = 2 and γ = 1.5, then the energy adjustment factor α(50) ≈ 0.418. Based on the energy adjustment factor α(50) ≈ 0.418, it can be concluded that the denoising intensity of this frame should be moderate. At this time, the denoising intensity of the filter can be adjusted according to this factor. Because α(50) is small, it indicates that the noise is relatively weak, so the denoising intensity does not need to be very strong. The gain of the filter can be kept at around 1.2, and the bandwidth can also be kept moderate. In other frames, the energy difference ratio R(t) will change according to the changes in noise energy and effective audio energy, thus causing the energy adjustment factor to change. If the noise energy of a certain frame is relatively large, resulting in a large R(t), such as R(t) = 1.5, then α(t) will increase, the gain coefficient of the filter will increase accordingly, and the bandwidth may also decrease to strengthen the denoising process and ensure that the noise component is effectively removed.

[0046] Step S105: Based on the amplitude spectrum and phase spectrum of the denoised karaoke audio signal, determine the lost phase information, calculate the envelope in combination with the amplitude spectrum, use linear interpolation to interpolate the phase of the missing region, and compensate for the time delay by shifting the time-frequency matrix.

[0047] By comparing the amplitude and phase spectra of the denoised karaoke audio signal, it is determined whether any phase information is lost. If phase information is lost, the frequency range of the lost phase is determined by examining the missing regions in the phase spectrum. At each frequency point, the instantaneous phase information of the signal is obtained by combining the logarithm of the amplitude spectrum with the phase spectrum, and the envelope of the signal is calculated based on the amplitude spectrum. Based on the known instantaneous phase and envelope information of adjacent frequency points, a linear interpolation method is used to interpolate the missing phase regions to recover the lost phase information. Based on the recovered lost phase information and the amplitude spectrum of the denoised karaoke audio signal, the time-frequency matrix of the repaired karaoke audio signal is reconstructed. Based on the time-frequency matrix of the repaired karaoke audio signal, a dynamic time warping algorithm is used to match the time axis of the audio signal to calculate the time delay between the input and output of the karaoke audio signal. Based on the time delay calculation results, the time column in the time-frequency matrix of the repaired karaoke audio signal is shifted to obtain the time-frequency matrix of the delay-compensated karaoke audio signal. The time-frequency matrix is ​​converted back to the time domain signal using the inverse short-time Fourier transform to obtain the output signal of the karaoke audio.

[0048] For example, in the denoised audio signal, some frequency components of the phase spectrum may be missing. This could be because the phase information was destroyed during the denoising process, or the noise removal algorithm lost phase information at some frequency points, such as missing phase spectra in the 1000Hz and 1500Hz bands. Therefore, it is necessary to repair this missing phase information. By examining the denoised phase spectrum, it is determined that frequency components between 1000Hz and 1500Hz are missing in the phase spectrum. To recover the phase information in these frequency ranges, it is necessary to combine the amplitude spectrum and phase spectrum to calculate the instantaneous phase and perform interpolation. By combining the logarithm of the amplitude spectrum with the phase spectrum, the instantaneous phase at each frequency point is obtained. For example, for the amplitude spectra at 1000Hz and 1500Hz, if their amplitude values ​​are 2.0 and 1.8 respectively, their logarithmic values ​​are log(2.0)≈0.3010 and log(1.8)≈0.2553. Calculating the instantaneous phase at these frequency points, the instantaneous phase at 1000Hz is 2.5 rad, and the instantaneous phase at 1500Hz is 1.8 rad. The envelope of the signal is calculated based on the amplitude spectrum, and the envelope values ​​for the 1000Hz and 1500Hz frequency bands are 1.2 and 1.0 respectively. Because phase information between 1000Hz and 1500Hz is missing, the instantaneous phase of 1000Hz is 2.5rad, and the instantaneous phase of 1500Hz is 1.8rad. Phase values ​​between these two frequencies are obtained through linear interpolation. Therefore, for 1200Hz, located between 1000Hz and 1500Hz, the phase value of the 1200Hz frequency band is calculated using the linear interpolation formula to be 2.22rad. Phase information for other missing frequency bands is recovered in a similar manner. After recovering the lost phase information, the repaired phase spectrum is combined with the amplitude spectrum to reconstruct the repaired time-frequency matrix. The time delay between the denoised audio signal and the original audio signal is calculated using a dynamic time warping algorithm, yielding a delay of 20ms. Based on this calculated delay, the time column of the repaired time-frequency matrix is ​​shifted backward by 20 milliseconds to eliminate the delay caused by the denoising process. By using inverse short-time Fourier transform to convert the time-frequency matrix after delay compensation back to the time domain signal, a karaoke audio output signal with noise removed and phase information restored can be obtained.

[0049] Step S106: Based on user feedback data, determine the noise reduction and audio restoration effects of the karaoke audio signal, and formulate and implement an improved karaoke audio signal processing plan.

[0050] By setting up a feedback mechanism, user feedback data on sound quality, audio latency, vocal clarity, and accompaniment synchronization after their karaoke experience is obtained to determine the denoising and audio restoration effects of the karaoke audio signal. If the denoising or audio restoration effects of the karaoke audio signal do not meet the preset standards, an improvement plan for the karaoke audio signal processing is formulated and implemented based on the user feedback data until both the denoising and audio restoration effects of the karaoke audio signal meet the preset standards. The improvement plan includes, but is not limited to, adjusting filter parameters, optimizing phase interpolation methods to fill in lost phase information, and readjusting latency compensation through dynamic time warping.

[0051] For example, a feedback mechanism is set up to collect user feedback data after the karaoke experience to evaluate the noise reduction and audio restoration effects. User feedback data includes a 3-star rating for sound quality, a 50ms audio latency, a 4-star rating for vocal clarity, and a 2-star rating for accompaniment-vocal synchronization. Based on this feedback data, it can be preliminarily determined that although the noise reduction effect of the audio signal has improved, the audio latency and accompaniment synchronization have not yet reached the preset standard of 3 stars and require further improvement. Based on user feedback, an improved processing plan for the karaoke audio signal is developed. Regarding the noise reduction effect, users reported that noise still exists, so the filter parameters need to be adjusted to further improve the noise reduction effect. The bandwidth of the currently used filter may be too wide, causing some effective audio components, such as the high-frequency part of the vocals, to be mistakenly removed. Based on user feedback, the filter bandwidth is reduced to 150Hz to focus more on removing low-frequency noise while maintaining high-frequency sound quality. In addition, the gain coefficient needs to be dynamically adjusted according to the energy adjustment factor. When noise is significant, the gain coefficient is increased to enhance the noise reduction effect. When noise is weak, the gain coefficient is reduced to ensure that audio quality is not compromised. Regarding audio restoration, user feedback indicated occasional blurring in high-frequency vocals, suggesting a loss of phase information, especially during noise removal. To recover this lost phase information, the phase interpolation method was optimized. By analyzing the amplitude and phase spectra after denoising, a phase information gap between 1000Hz and 1500Hz was identified. The lost phase was recovered through linear interpolation, and combined with the logarithm of the amplitude spectrum, the phase in the missing region was restored to a suitable range, ensuring clarity in high-frequency frequencies. Due to user feedback regarding long audio delays, a dynamic time warping algorithm was used to re-optimize the audio signal's delay compensation. The calculated input-output delay was 50ms, indicating a significant time difference between the denoised and original audio signals. Based on this, the time-frequency matrix of the denoised audio signal was shifted forward by 50ms to ensure synchronization between the audio and vocals. After implementing the above improvements, the audio signal was reprocessed, and user feedback was collected again. Based on the new user feedback, it was determined that both the noise reduction and audio restoration effects met the preset performance standards. By adjusting filter parameters, optimizing the phase interpolation method, and dynamically adjusting delay compensation using time warping, the noise, phase loss, audio delay, and accompaniment synchronization issues in the audio signal were successfully resolved, significantly improving the quality of the karaoke audio signal.

[0052] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A karaoke audio processing method based on human voice separation and restoration, characterized in that, The method includes: The audio acquisition module acquires the karaoke audio signal, and the short-time Fourier transform algorithm is used to transform the karaoke audio signal from the time domain to the frequency domain to obtain the time-frequency matrix of the karaoke audio signal, and to determine whether there are noise components in the karaoke audio signal. Based on the time-frequency matrix of the karaoke audio signal, nonnegative matrix decomposition is used to obtain the basis matrix and activation matrix, identify the noise basis matrix and effective audio basis matrix, calculate the overlap coefficient of noise and effective audio in different frequency bands, and determine the crossover frequency band. Based on the noise basis matrix and the corresponding activation matrix, the energy value, zero-crossing rate, spectral flatness and statistical characteristics of each frame of the noise signal are determined, and transient noise and steady-state noise are identified. Based on the temporal envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal, an adaptive filtering algorithm is used to design a filter to filter the noise of each frame in real time, thereby obtaining the denoised karaoke audio signal. By comparing the amplitude spectrum and phase spectrum of the denoised karaoke audio signal, the lost phase information is determined. The envelope is calculated in combination with the amplitude spectrum, the phase of the missing region is interpolated using linear interpolation, and the time delay is compensated by shifting the time-frequency matrix. Based on user feedback data, we assess the noise reduction and audio restoration effects of karaoke audio signals, and formulate and implement improvement plans for karaoke audio signal processing.

2. The method according to claim 1, wherein, The process involves acquiring karaoke audio signals through an audio acquisition module, transforming the karaoke audio signals from the time domain to the frequency domain using a short-time Fourier transform algorithm, obtaining the time-frequency matrix of the karaoke audio signals, and determining whether there are noise components in the karaoke audio signals, including: The audio acquisition module acquires the karaoke audio signal. Based on a preset frame length and frame shift, the audio signal is divided into several overlapping short-time frame sequences. For each frame, a Hamming window is applied, and a short-time Fourier transform algorithm is used to transform the karaoke audio signal from the time domain to the frequency domain. The spectrum of each frame is calculated, and the time-frequency matrix of the karaoke audio signal, including the amplitude spectrum and phase spectrum, is obtained. Based on the amplitude spectrum of the karaoke audio signal, the power spectrum and spectral flatness of each frame are calculated to determine the energy distribution of each frequency band. Based on the energy distribution and spectral flatness of each frequency band, a support vector machine algorithm is used to train the model and construct a noise component discrimination model to determine whether there are noise components in the karaoke audio signal.

3. The method according to claim 1, wherein, The process involves obtaining the basis matrix and activation matrix using nonnegative matrix decomposition based on the time-frequency matrix of the karaoke audio signal, identifying the noise basis matrix and the effective audio basis matrix, calculating the overlap coefficient of noise and effective audio in different frequency bands, and determining the crossover frequency band, including: Based on the time-frequency matrix of the karaoke audio signal, a nonnegative matrix decomposition of the amplitude spectrum is used. The number of basis functions and iterations are set, and KL divergence is used as the loss function to decompose the amplitude spectrum of the karaoke audio signal. The basis matrix and activation matrix are alternately optimized using a multiplication update rule. The basis matrix represents the fundamental frequency components of the audio signal, and the activation matrix represents the activation level of the fundamental frequency components in each frame. Based on the column vectors of the basis matrix, K-means clustering is used to perform cluster analysis to identify the noise basis matrix and the effective audio basis matrix. The effective audio includes vocals and accompaniment. Based on the activation matrices corresponding to the noise basis matrix and the effective audio basis matrix, a cosine similarity calculation method is used to calculate the similarity between the activation values ​​of the noise component and the activation values ​​of the effective audio component in each frequency band, determining the overlap coefficient between noise and effective audio in different frequency bands. If the overlap coefficient is greater than a preset threshold, the frequency band is determined to be an intersecting frequency band.

4. The method according to claim 1, wherein, The process of determining the energy value, zero-crossing rate, spectral flatness, and statistical characteristics of each frame of the noise signal based on the noise basis matrix and the corresponding activation matrix, and identifying transient and steady-state noise, includes: Based on the noise basis matrix and the corresponding activation matrix, the short-time energy calculation formula is used. The energy value of each frame of the noise signal is obtained, and the temporal envelope characteristics of the noise signal are determined. Let be the audio signal energy at time t. Let F be the amplitude spectrum value at time t and frequency f, where F is the frequency range, and the time-domain envelope characteristic is the energy fluctuation amplitude of the noise signal. Based on the noise basis matrix, calculate the zero-crossing rate and spectral flatness of each frame of noise signal. The zero-crossing rate is the number of times the signal crosses zero points per unit time. Calculate the energy value difference of each frame of noise signal. If there is a short-time frame sequence with an energy value difference greater than a preset difference threshold or a zero-crossing rate greater than a preset zero-crossing rate threshold, then mark the noise of that frame as transient noise; otherwise, mark it as steady-state noise. By comparing the energy value of each frame with the energy values ​​of the preceding and following frames, determine the duration and intensity of transient noise. Use statistical methods to calculate the mean, variance, kurtosis, and skewness of the noise signal energy to determine the statistical characteristics of the noise signal.

5. The method according to claim 1, wherein, The process involves designing a filter using an adaptive filtering algorithm based on the time-domain envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal. This filter performs real-time filtering on the noise in each frame to obtain the denoised karaoke audio signal, including: Based on the temporal envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal, an adaptive filtering algorithm is used to design a filter and determine the initial parameters of the filter. Real-time filtering is performed on the noise of each frame. The initial parameters of the filter include, but are not limited to, the filter length, bandwidth, and gain coefficient. If a short frame of transient noise is identified in the noise signal, an energy adjustment factor is calculated based on the energy of the noise signal and the effective audio in that frame, and the denoising intensity of each frame's audio signal is adjusted based on the energy adjustment factor. If the energy adjustment factor is greater than a preset threshold, the gain coefficient of the filter is increased or the bandwidth is reduced based on the energy adjustment factor. The filter parameters are dynamically adjusted using the temporal envelope characteristics, spectral flatness, statistical properties, basis matrix, cross-band, and overlap coefficient of the noise signal in the real-time acquired karaoke audio signal. The filter removes the noise components from the original karaoke audio signal, resulting in a denoised karaoke audio signal, and a time-frequency matrix of the denoised karaoke audio signal is generated.

6. The method according to claim 5, wherein, If a short frame of transient noise is identified in the noise signal, an energy adjustment factor is calculated based on the noise signal and the energy of the effective audio in that frame, and the denoising intensity of each frame's audio signal is adjusted based on the energy adjustment factor, including: If a short frame of transient noise is identified, the energy difference ratio between the noise source and the valid audio source is calculated based on the energy of the noise signal and the valid audio signal in that frame. ,in, It is the energy of the noise signal. It is the energy of the effective audio source; based on the energy difference ratio between the noise source and the effective audio source, the energy adjustment factor formula is used. Calculate the energy regulation factor , where β and γ are adjustment parameters used to control the sensitivity of noise and audio balance, obtained by fitting historical data; the denoising intensity of each frame of audio signal is dynamically adjusted according to the energy adjustment factor.

7. The method according to claim 1, wherein, The process involves comparing the amplitude and phase spectra of the denoised karaoke audio signal to determine the lost phase information, calculating the envelope based on the amplitude spectrum, using linear interpolation to interpolate the phase of the missing region, and compensating for time delay by shifting the time-frequency matrix. This includes: By comparing the amplitude and phase spectra of the denoised karaoke audio signal, it is determined whether there is any lost phase information. If phase information is lost, the frequency range of the lost phase is determined by examining the missing regions in the phase spectrum. At each frequency point, the instantaneous phase information of the signal is obtained by combining the logarithm of the amplitude spectrum with the phase spectrum, and the envelope of the signal is calculated based on the amplitude spectrum. Based on the known instantaneous phase and envelope information of adjacent frequency points, a linear interpolation method is used to interpolate the missing phase region to recover the lost phase information. Phase information, combined with the amplitude spectrum of the denoised karaoke audio signal, is used to reconstruct the time-frequency matrix of the repaired karaoke audio signal. Based on the time-frequency matrix of the repaired karaoke audio signal, a dynamic time warping algorithm is used to match the time axis of the audio signal and calculate the time delay between the input and output of the karaoke audio signal. Based on the time delay calculation result, the time column in the time-frequency matrix of the repaired karaoke audio signal is shifted to obtain the time-frequency matrix of the delay-compensated karaoke audio signal. The inverse short-time Fourier transform is used to convert the time-frequency matrix back to the time domain signal to obtain the output signal of the karaoke audio signal.

8. The method according to claim 1, wherein, The process of judging the noise reduction and audio restoration effects of karaoke audio signals based on user feedback data, and formulating and implementing an improved karaoke audio signal processing plan includes: By setting up a feedback mechanism, user feedback data on sound quality, audio latency, vocal clarity, and accompaniment synchronization after the user's karaoke experience is obtained to determine the noise reduction and audio restoration effects of the karaoke audio signal. If the noise reduction or audio restoration effect of the karaoke audio signal does not meet the preset effect standard, an improvement plan for karaoke audio signal processing is formulated and implemented based on the user feedback data until the noise reduction and audio restoration effects of the karaoke audio signal both meet the preset effect standard. The improvement plan includes, but is not limited to, adjusting filter parameters, optimizing phase interpolation methods to fill in lost phase information, and readjusting latency compensation through dynamic time warping.

Images