An audio signal processing method for a sound equipment based on artificial intelligence

Abstract

The application relates to the technical field of audio signal processing, and discloses an audio signal processing method for a sound equipment based on artificial intelligence, which comprises the following steps: step S1, acquiring an audio signal, performing short-time Fourier transform on the audio signal, and converting the audio signal into a time-frequency domain representation; step S2, performing normalization processing on the transformed audio signal; step S3, inputting the normalized audio signal into a variational autoencoder, and extracting a low-dimensional latent feature vector; and step S4, inputting the feature vector extracted by the variational autoencoder into a diffusion model, and removing noise by using an inverse diffusion process. Through the personalized audio enhancement technical scheme based on user hearing sensation feature modeling, the audio frequency response characteristics can be dynamically adjusted according to user preferences, the sound quality output conforming to the user hearing habits is realized, and compared with the scheme in the prior art that adopts a unified audio enhancement model, the limitations that the personalized demand cannot be met and the sound quality adjustment is fixed are overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio signal processing, in particular to an audio signal processing method for a sound equipment based on artificial intelligence. BACKGROUND

[0002] With the wide popularity of intelligent sound equipment, the importance of audio signal processing technology is increasingly prominent. Modern sound equipment needs to have high-quality audio playback capability and be able to adapt to complex acoustic environments, perform voice enhancement, noise suppression, personalized sound effect adjustment, and sound source separation, etc. tasks. However, traditional audio processing methods are mostly based on fixed parameter filters and signal processing algorithms, which cannot meet the current user's demand for low delay, high fidelity, and personalization. In recent years, the introduction of artificial intelligence technology provides a new way for audio signal processing, especially the development of deep learning and adaptive algorithms, which enables accurate and efficient completion of audio enhancement, noise reduction, separation, and compression tasks. However, existing artificial intelligence audio processing technology still has many limitations in terms of delay, flexibility, and computational resource optimization, which restricts the widespread landing of practical applications.

[0003] Most current voice enhancement algorithms are based on fixed filters or deep learning models, which have high computational complexity and cannot meet the low delay requirement. In particular, in real-time interactive scenarios such as intelligent assistants and online conference systems, existing methods usually require a large amount of computing resources, resulting in tens of milliseconds of delay in voice processing, which affects user experience. In addition, traditional time-frequency domain transformation-based noise reduction methods react slowly when the noise environment changes suddenly, and cannot quickly adjust the filtering strategy, resulting in unstable noise suppression effect.

[0004] Traditional sound equipment mostly adopts preset sound effect enhancement modes, such as "popular", "classical", "heavy bass", etc. fixed equalizer adjustment methods, lacking intelligent adaptation to individual user listening preferences. Some existing personalized audio enhancement methods based on machine learning can adjust the sound effect to some extent according to user historical preferences, but the model training cost is high, and the personalized adjustment granularity is limited, which cannot provide continuous optimization in complex environments. In addition, the generalization ability of the personalized enhancement model is weak, and the performance on different users and devices is different, affecting the consistency of experience.

[0005] Existing voice separation methods mostly use traditional blind source separation technology or deep learning methods based on single-channel, which are prone to sound source leakage and aliasing problems when processing complex sound fields. For example, in a multi-person conversation scene or a complex music environment, existing methods cannot accurately distinguish the spatial information of different sound sources, resulting in a decrease in the intelligibility of the target voice. In addition, sound field optimization technology usually relies on preset filter parameters and cannot adaptively adjust according to different spatial environments, resulting in insufficient sound image positioning and spatial hierarchy, affecting immersive audio experience.

[0006] Traditional audio compression encoding methods, such as MP3, AAC and Opus, mostly use perceptual coding technology based on the auditory masking effect of the human ear to reduce the bit rate. However, in the low bit rate transmission scene, the existing algorithm has obvious loss of sound quality, especially in high dynamic range or complex tone audio, which is prone to high frequency loss and artifacts. In addition, the current AI audio encoding and decoding technology based on neural network has improved the sound quality, but the calculation complexity is high, and the device computing power is required, which cannot be efficiently run in embedded devices or low power consumption scenes, limiting the universality of practical application.

[0007] Therefore, the person skilled in the art provides an audio device audio signal processing method based on artificial intelligence to solve the problems raised in the above. SUMMARY

[0008] In view of the deficiencies of the prior art, the present application provides an audio device audio signal processing method based on artificial intelligence to solve the problems raised in the above background.

[0009] To achieve the above purpose, the present application is implemented by the following technical scheme: an audio device audio signal processing method based on artificial intelligence, comprising the following steps:

[0010] Step S1, acquiring an audio signal, performing short-time Fourier transform on the audio signal, and converting it into a time-frequency domain representation;

[0011] Step S2, normalizing the transformed audio signal;

[0012] Step S3, inputting the normalized audio signal into a variational autoencoder to extract a low-dimensional latent feature vector;

[0013] Step S4, inputting the feature vector extracted by the variational autoencoder into a diffusion model, and removing noise using an inverse diffusion process;

[0014] Step S5, performing inverse short-time Fourier transform on the denoised audio signal to restore the time domain signal;

[0015] Step S6, post-processing the restored time domain signal;

[0016] Step S7, outputting the enhanced audio signal.

[0017] Preferably, the step S1 further comprises:

[0018] Step 1.1, collecting the original audio signal: acquiring continuous audio signals through a microphone array and a single-channel microphone, and the sampled discrete signal is represented as: x[n]=x(nT s ),

[0019] Wherein, T sis the sampling period, x[n] is the discrete audio signal after sampling, and n is the discrete time index;

[0020] Step 1.2, calculating the spectrum signal: using a set frame length N to perform a short-time Fourier transform, the signal x[n] is framed, and each frame signal is represented as: x m [k] = x[k + mR], 0≤k<N,

[0021] wherein x m [k] is the signal sample in the mth frame, k is the sampling point index within the frame, m is the frame index, R is the frame shift length, R is the sampling point index within the frame, and the window w[k] is added to each frame signal, and a short-time Fourier transform is calculated:

[0022] wherein X m [f] is the spectrum of the mth frame, f is the frequency index, j is the imaginary unit, N is the frame length, e -j2πfk / N is the oscillation in the frequency domain, π is a constant, and w[k] is a window function;

[0023] Step 1.3, adjusting the window function: according to the statistical characteristics of the audio signal, the window function w[k] is optimized. First, the short-time energy of the signal is calculated:

[0024] Then, the short-time zero-crossing rate of the signal is calculated:

[0025] wherein sgn is a sign function, E m is the short-time energy of the mth frame, Z m is the short-time zero-crossing rate of the mth frame, E m and Z m are combined to determine the parameters of the window function, such as:

[0026]

[0027] wherein α is a dynamically adjusted parameter, and α is dynamically adjusted according to the statistical characteristics of E m and Z m .

[0028] Preferably, the step S2 further comprises:

[0029] Step 2.1, calculating the amplitude spectrum of the input audio signal: the short-time Fourier transform of the audio signal obtained in step S1 is represented as: X(f, t) = A(f, t)e jθ(f,t)

[0030] wherein X(f, t) is a frequency domain complex signal, f is a frequency index, t is a time frame index, A(f, t) is an amplitude spectrum, θ(f, t) is phase information, and j is an imaginary unit.

[0031] On this basis, the amplitude spectrum A(f, t) is obtained by calculating the modulus value of the complex signal X(f, t):

[0032] A(f, t) = |X(f, t)|,

[0033] Step 2.2, calculating the global mean and standard deviation: calculating the amplitude spectrum mean and standard deviation of all time frames:

[0034]

[0035] where T is the total number of time frames, F is the total number of frequency points, μ A is the global amplitude mean of the overall audio signal, σ A is the global standard deviation of the overall audio signal;

[0036] Step 2.3, normalizing the amplitude spectrum: standardizing the amplitude spectrum:

[0037]

[0038] where A is the normalized amplitude spectrum, and ∈ is a numerical stability parameter;

[0039] Step 2.4, amplitude limiting: the normalized amplitude spectrum is limited to prevent extreme values from affecting the model instability:

[0040]

[0041] where A is the amplitude-limited amplitude spectrum, and C max is the amplitude limiting threshold;

[0042] Step 2.5, normalized frequency spectrum recovery: on the basis of the normalized amplitude spectrum, the original phase information θ(f, t a ) is kept unchanged, and the normalized frequency domain signal is recovered:

[0043]

[0044] where X is the normalized complex frequency domain signal;

[0045] Step 2.6, output the normalized frequency spectrum signal: the normalized signal X is passed to step S3 for variational autoencoder feature extraction.

[0046] Preferably, the step S3 further comprises:

[0047] Step 3.1, Building the Variational Autoencoder: The variational autoencoder encodes the input data into a low-dimensional representation through a probabilistic model. First, the encoder network is built The normalized spectral signal is mapped to the distribution of latent variables Z.

[0048] The encoder consists of multiple groups of convolutional layers, which compute the mean and variance of the latent variables:

[0049]

[0050] where μ Z (t) is the mean of the latent variable Z, σ μ (t) is the variance of the latent variable Z, f σ and f Z are neural network mapping functions, and φ is the parameter. is the normalized complex frequency domain signal, f is the frequency index, and t is the time frame index.

[0051] The latent variable Z is sampled using the reparameterization trick:

[0052] Z(t) = μ Z (t) · β,

[0053] where Z(t a ) is the sampled latent variable, and β is a noise term from a standard normal distribution.

[0054] Step 3.2, Calculating the Variational Autoencoder Objective Function: The objective function of the variational autoencoder consists of the reconstruction error and the KL divergence term:

[0055]

[0056] where is the spectral signal generated by the decoder, L VAE is the total loss function of the variational autoencoder.

[0057] is the KL divergence between the latent distribution and the standard normal distribution p(Z), and λ is the regularization coefficient.

[0058] The KL divergence is further expanded as:

[0059]

[0060] where D KL is the KL divergence loss, D is the dimension of the latent space, and are the mean and variance of the latent variable in dimension d.

[0061] Step 3.3, low-dimensional feature space regularization: Introduce a manifold regularization method to constrain the continuity of the learned features in a low-dimensional space, using a local neighborhood smoothness loss:

[0062]

[0063] where N(t) is the neighborhood of time frame t, w t′ is the neighborhood weighting coefficient, L MR is the manifold regularization loss;

[0064] Finally, the optimization objective function of the variational autoencoder becomes:

[0065] L total = L VAE + γL MR ,

[0066] where L total is the final optimization objective, and γ is a hyperparameter that controls the strength of the manifold regularization;

[0067] Step 3.4, recover the preliminary denoised spectrum through the variational autoencoder decoder: Use the decoder network to map the latent variable Z back to the time-frequency spectrum space, recovering the preliminary denoised audio signal:

[0068]

[0069] where g θ is the decoder network trained by parameters θ, is the denoised audio spectrum generated by the variational autoencoder;

[0070] The recovered spectrum signal is passed to step S4 for denoising processing by the diffusion model to further remove residual noise and optimize sound quality.

[0071] Preferably, the step S4 further comprises:

[0072] Step 4.1, construct the forward diffusion process: The forward process of the diffusion model gradually adds noise to make the input signal tend to a standard Gaussian distribution;

[0073] Given the preliminary denoised spectrum signal The forward diffusion process is defined as follows:

[0074]

[0075] where Y k is the noise version signal generated at the k-th step, Y k-1 is the noise version signal generated at the k-1-th step, and α kis the attenuation coefficient, I is the identity matrix, and k is the diffusion time step;

[0076] By repeatedly applying the forward diffusion process, the final fully noisy signal Y is obtained k :

[0077] q(Y K |Y0) = N(Y K ; 0, I), where Y0is the initial input;

[0078] Step 4.2, Calculate the noise estimate of the inverse diffusion process: The purpose of the inverse diffusion process is to remove noise and gradually restore the signal from noise to high-quality audio signal;

[0079] Parameterize the inverse diffusion process with a neural network, and use the estimated noise model∈ θ (Y K |Y) to calculate the mean and variance of the denoised signal:

[0080]

[0081] where μ θ (Y k ,k) is the estimated mean of the denoising network, is the variance in the denoising process, which controls the uncertainty;

[0082] The estimated mean μ θ (Y k ,k) is calculated by the following formula:

[0083]

[0084] where∈ θ (Y k ,k) is the noise component estimated by the neural network;

[0085] Step 4.3, Calculate the loss function of the diffusion model: The optimization goal of the diffusion model is to minimize the mean square error between the real noise∈ and the estimated noise∈ θ (Y k ,k):

[0086]

[0087] where L DM is the loss function of the diffusion model, E is the expectation calculation for all samples, and∈ is the real noise;

[0088] By minimizing L DM , the neural network is trained to correctly estimate the noise component at different time steps k to achieve denoising;

[0089] Step 4.4, variable step size inverse diffusion process optimization: a variable step size diffusion strategy is adopted to dynamically adjust the step size ak:

[0090] Wherein, τ is a dynamic factor for controlling the change of step size, K is the total number of diffusion steps;

[0091] Step 4.5, output the denoised audio signal: after diffusion model denoising, the optimized audio spectrum is obtained

[0092] Wherein, Y1 is the noise version signal generated in the first step;

[0093] The denoised spectrum signal is transmitted to step S5 for inverse short-time Fourier transform to restore the time domain audio signal for subsequent audio enhancement and playback.

[0094] Preferably, the step S5 further comprises:

[0095] Step 5.1, setting the inverse short-time Fourier transform parameters: before performing the inverse short-time Fourier transform, set the necessary inverse short-time Fourier transform parameters;

[0096] Step 5.2, inverse short-time Fourier transform calculation: after setting the parameters of the inverse short-time Fourier transform, perform the inverse transform to restore the frequency domain signal to the time domain signal y(t) using the following formula:

[0097]

[0098] Wherein, y(t) is the restored time domain audio signal, is the restored denoised frequency domain signal, w(t k ) is the window function, f k is the frequency point, t k is the time point, j is the imaginary unit, π is the constant, t is the sampling point of time domain;

[0099] Step 5.3, reconstruct the audio signal: after the inverse short-time Fourier transform is completed, the final time domain audio signal y(t) is obtained by synthesizing each overlapping part;

[0100] Step 5.4, output the final audio signal: the final time domain signal y(t) is output to the audio player for playback and saving.

[0101] Preferably, the step S6 further comprises:

[0102] Step 6.1, calculating short-time power spectrum of time-domain audio signal: before adaptive filtering, short-time power spectrum calculation is performed on the time-domain signal y(t) recovered in step S5 to analyze the frequency distribution and energy characteristics of the signal, and the short-time power spectrum is defined as follows:

[0103]

[0104] where P y (f m ,t n ) is the short-time power spectrum at the t n th time frame and the f m th frequency point, M is the length of the window function, w k is the windowing function, f m is the frequency index, t n is the time frame index, j is the imaginary unit, π is a constant, and t is the sampling point in the time domain.

[0105] Step 6.2, estimating noise power spectrum: assuming that the noise signal n(t) follows a stationary Gaussian distribution, the power spectrum is estimated by the method of time averaging:

[0106]

[0107] where is the estimated noise power spectrum, λ is the smoothing factor, and P y (f m ,t n ) is the short-time power spectrum.

[0108] Step 6.3, calculating adaptive filtering gain: after obtaining the short-time power spectrum P y (f m ,t n ) of the audio signal and the estimated noise power spectrum , the Wiener filtering gain G(f m ,t n ) is calculated to perform noise reduction processing:

[0109]

[0110] where G(f m ,t n ) is the Wiener filtering gain.

[0111] γ is the lower limit of the gain, which prevents the filter from excessively suppressing the audio signal, P y (f m ,t n ) is the short-time power spectrum, is the noise power spectrum.

[0112] Step 6.4, computing the enhanced audio signal: using the computed Wiener filter gain G(f m ,t n ), the audio signal y(t) outputted in step S5 is enhanced, and the enhanced signal spectrum is computed as follows:

[0113]

[0114] wherein, is the enhanced signal spectrum, G(f m ,t n ) is the computed Wiener filter gain;

[0115] The enhanced signal spectrum is then converted back to a time-domain signal by inverse Fourier transform:

[0116]

[0117] wherein, is the enhanced time-domain audio signal;

[0118] Step 6.5, outputting the enhanced audio signal: the Wiener filter enhanced time-domain signal is outputted as the output of step S6, and is used for subsequent playing, storage and further processing:

[0119]

[0120] wherein, y enhanced (t) is the final enhanced audio signal.

[0121] Preferably, the AI-based audio device audio signal processing method adopts model pruning and quantization techniques to reduce the consumption of computing resources, wherein the pruning process is achieved by optimizing the following objective function:

[0122] wherein, is the optimized weight matrix, W is the weight of the original neural network, is the weight retained after pruning, Q is the quantization step, is the optimization solving process,

[0123] is the squared error between the original weight W and the quantized weight .

[0124] A terminal device comprising a memory, a processor and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the AI-based audio device audio signal processing method.

[0125] A computer-readable storage medium stores a computer program, which, when executed in a computing device, is used to implement an artificial intelligence-based audio device audio signal processing method.

[0126] The present application provides an artificial intelligence-based audio device audio signal processing method. It has the following

[0127] Advantages:

[0128] 1. The adaptive multi-level feature extraction and real-time neural network noise reduction technology scheme is adopted, which can significantly reduce the speech enhancement delay and improve the speech intelligibility. Compared with the scheme based on the fixed parameter filter in the prior art, the high delay of the speech enhancement process and the poor enhancement effect in the complex noise environment are solved.

[0129] 2. The personalized audio enhancement technology scheme based on user hearing characteristics modeling can dynamically adjust the audio frequency response characteristics according to user preferences, realize the sound quality output conforming to user hearing habits, and overcome the limitations of not meeting personalized needs and sound quality adjustment fixation compared with the scheme using a unified audio enhancement model in the prior art.

[0130] 3. The multi-channel spatial perception and deep learning combined end-to-end speech separation technology scheme can effectively distinguish and enhance the target speech, and optimize the direction and level of the spatial sound field. Compared with the serial scheme in the prior art which relies on independent sound source separation and sound field processing, the sound source aliasing problem is avoided, and the separation accuracy and sound field realism are improved.

[0131] 4. The audio compression and decoding technology scheme combining variational autoencoder and quantization neural network is introduced, which can greatly reduce the data transmission bandwidth requirement while ensuring the sound quality. Compared with the traditional compression method based on linear predictive coding, the problem of low compression ratio of high-fidelity audio and sound quality degradation in complex acoustic environment is solved. BRIEF DESCRIPTION OF DRAWINGS

[0132] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0133] In order to enable those skilled in the art to understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, other embodiments obtained by those skilled in the art without creative labor should be within the scope of the present application.

[0134] The present application will be described in detail below with reference to the drawings:

[0135] Embodiments:

[0136] Please refer to the accompanying Figure 1 , the embodiment of the application provides a kind of audio signal processing method of audio equipment based on artificial intelligence, comprising the following steps:

[0137] Step S1, obtaining audio signal, short-time Fourier transform is carried out to audio signal, and conversion is carried out into time-frequency domain representation;

[0138] Step 1.1, original audio signal is collected: continuous audio signal is obtained by microphone array and single-channel microphone, and the discrete signal after sampling is represented as: x [n]=x (nT s ),

[0139] Wherein, T s For sampling period, x [n] is the discrete audio signal after sampling, and n is discrete time index;

[0140] Step 1.2, calculate spectrum signal: short-time Fourier transform is carried out using setting frame length N, frame signal is carried out to signal x [n], and each frame signal is represented as: x m [k]=x [k+mR], 0≤k<N,

[0141] Wherein, x m [k] is the signal sample in the mth frame, k is the sampling point index within frame, m is frame index, R is frame shift length, R is the sampling point index within frame, and each frame signal is windowed w [k] after, and short-time Fourier transform is calculated:

[0142] Wherein, X m [f] is the spectrum of the mth frame, f is frequency index, j is imaginary unit, N is frame length, e -j2πfk / N Is the oscillation in frequency domain, π is constant, and w [k] is window function;

[0143] Step 1.3, adjust window function: according to the statistical characteristics of audio signal, optimize window function w [k], first, the short-time energy of signal is calculated:

[0144] Then, the short-time zero-crossing rate of signal is calculated:

[0145] Wherein, sgn is sign function, E m It is the short-time energy of the mth frame, Z m It is the short-time zero-crossing rate of the mth frame, and the parameters of window function are determined in combination with E m And Z m , such as:

[0146]

[0147] where a is a dynamically adjusted parameter, a is dynamically adjusted according to E m and Z m statistical characteristics of the input audio signal;

[0148] Step S2, normalizing the transformed audio signal;

[0149] Step 2.1, calculating the amplitude spectrum of the input audio signal: the short-time Fourier transform transformed audio signal obtained in step S1 is represented as: X(f, t) = A(f, t)e jθ(f,t)

[0150] where X(f, t) is a frequency domain complex signal, f is a frequency index, t is a time frame index, A(f, t) is an amplitude spectrum, θ(f, t) is phase information, and j is an imaginary unit;

[0151] On this basis, the amplitude spectrum A(f, t) is obtained by calculating the modulus of the complex signal X(f, t):

[0152] A(f, t) = |X(f, t)|,

[0153] Step 2.2, calculating the global mean and standard deviation: calculating the amplitude spectrum mean and standard deviation of all time frames:

[0154]

[0155] where T is the total number of time frames, F is the total number of frequency points, μ A is the global amplitude mean of the overall audio signal, and σ A is the global standard deviation of the overall audio signal;

[0156] Step 2.3, normalizing the amplitude spectrum: normalizing the amplitude spectrum:

[0157]

[0158] where A is the normalized amplitude spectrum, and ∈ is a numerical stability parameter;

[0159] Step 2.4, amplitude limiting: the normalized amplitude spectrum is amplitude limited to prevent extreme values from affecting the model stability:

[0160]

[0161] where A is the amplitude limited amplitude spectrum, and C max is the amplitude limiting threshold;

[0162] Step 2.5, Normalized spectrum recovery: Based on the normalized amplitude spectrum, the original phase information θ(f, t a ) is kept unchanged, and the normalized frequency domain signal is recovered:

[0163]

[0164] wherein, is the normalized complex frequency domain signal;

[0165] Step 2.6, Output normalized spectrum signal: The normalized signal is passed to step S3 for variational autoencoder feature extraction;

[0166] Step S3, input normalized audio signal into variational autoencoder to extract low-dimensional latent feature vector;

[0167] Step 3.1, construct variational autoencoder: the variational autoencoder encodes the input data into low dimension through a probability model, and first establishes an encoder network to map the normalized spectrum signal to the distribution of latent variable Z;

[0168] The encoder is composed of multiple groups of convolutional layers, and the mean and variance of the latent variable are calculated:

[0169]

[0170] wherein, μ Z (t) is the mean of the latent variable Z, is the variance of the latent variable Z, f μ and f σ are neural network mapping functions, φ is a parameter, is the normalized complex frequency domain signal, f is the frequency index, and t is the time frame index;

[0171] The latent variable Z is sampled using the reparameterization trick:

[0172] Z(t) = μ Z (t) + σ Z (t) · β,

[0173] wherein, Z(t a ) is the sampled latent variable, and β is a noise term of standard normal distribution;

[0174] Step 3.2, calculate the variational autoencoder objective function: the objective function of the variational autoencoder is composed of the reconstruction error and the KL divergence term:

[0175]

[0176] wherein, is the decoder-generated spectrum signal, L VAE is the total loss function of the variational autoencoder;

[0177] is the latent distribution is the KL divergence between the standard normal distribution p(Z) and the latent distribution q(Z|X), and λ is the regularization coefficient;

[0178] The KL divergence is further expanded as:

[0179]

[0180] where D KL is the KL divergence loss, D is the latent space dimension, and are the mean and variance of the latent variable in dimension d;

[0181] Step 3.3, low-dimensional feature space regularization: Introduce a manifold regularization method to constrain the continuity of the features learned by the encoder in the low-dimensional space, using a local neighborhood smoothing loss:

[0182]

[0183] where N(t) is the neighborhood of time frame t, w t′ is the neighborhood weighting coefficient, L MR is the manifold regularization loss;

[0184] Finally, the optimization objective function of the variational autoencoder becomes:

[0185] L total = L VAE + γL MR ,

[0186] where L total is the final optimization objective, and γ is a hyperparameter that controls the strength of manifold regularization;

[0187] Step 3.4, recover the preliminary denoised spectrum through the variational autoencoder decoder: Use the decoder network to map the latent variable Z back to the time-frequency spectrum space and recover the preliminary denoised audio signal:

[0188]

[0189] where g θ is the decoder network trained by parameters θ, is the denoised audio spectrum generated by the variational autoencoder;

[0190] The restored spectral signal is passed to step S4 for denoising processing of the diffusion model to further remove residual noise and optimize sound quality;

[0191] Step S4, input the feature vector extracted by the variational autoencoder into the diffusion model, and remove noise by using an inverse diffusion process;

[0192] Step 4.1, construct a forward diffusion process: the forward process of the diffusion model gradually adds noise so that the input signal tends to a standard Gaussian distribution;

[0193] Given the preliminary denoised spectral signal The forward diffusion process is defined as follows:

[0194]

[0195] where Y k is the noise version signal generated at the kth step, Y k-1 is the noise version signal generated at the k-1th step, α k is the attenuation coefficient, I is the identity matrix, and k is the diffusion time step;

[0196] By repeatedly applying the forward diffusion process, the final completely noisy signal Y k is obtained:

[0197] q(Y K |Y0)=N(Y K ;0,I), where Y0 is the initial input;

[0198] Step 4.2, calculate the noise estimate of the inverse diffusion process: the purpose of the inverse diffusion process is to remove noise so that the signal gradually recovers from noise to high-quality audio signal;

[0199] Parameterize the inverse diffusion process using a neural network, and calculate the mean and variance of the denoised signal using the estimated noise model ∈ θ (Y K |Y):

[0200]

[0201] where μ θ (Y k ,k) is the estimated mean of the denoising network, is the variance in the denoising process, which controls the uncertainty;

[0202] The estimated mean μ θ (Y k ,k) is calculated by the following formula:

[0203]

[0204] where ∈ θ (Y k ,k) is the noise component estimated by the neural network;

[0205] Step 4.3, calculate the loss function of the diffusion model: the optimization goal of the diffusion model is to minimize the mean square error between the real noise ∈ and the estimated noise ∈ θ (Y k ,k) is the noise component estimated by the neural network;

[0206]

[0207] where L DM is the loss function of the diffusion model, E is the expectation calculation of all samples, ∈ is the real noise;

[0208] By minimizing L DM , the neural network is trained to correctly estimate the noise component at different time steps k, and denoising is achieved;

[0209] Step 4.4, variable step size inverse diffusion process optimization: a variable step size diffusion strategy is adopted to dynamically adjust the step size α k :

[0210] where τ is a dynamic factor that controls the change of the step size, and K is the total number of diffusion steps;

[0211] Step 4.5, output the denoised audio signal: after denoising by the diffusion model, the optimized audio spectrum

[0212] where Y1 is the noise version signal generated in the first step;

[0213] The denoised spectrum signal is passed to step S5 for inverse short-time Fourier transform to restore the time-domain audio signal for subsequent audio enhancement and playback;

[0214] Step S5, inverse short-time Fourier transform is performed on the denoised audio signal to restore the time-domain signal;

[0215] Step 5.1, set the inverse short-time Fourier transform parameters: before performing the inverse short-time Fourier transform, set the necessary inverse short-time Fourier transform parameters;

[0216] Step 5.2, inverse short-time Fourier transform calculation: after setting the parameters of the inverse short-time Fourier transform, perform the inverse transform to restore the frequency domain signal to the time domain signal y(t) using the following formula:

[0217]

[0218] where y(t) is the restored time-domain audio signal, for the restored de-noised frequency domain signal w(t k ) is a window function, f k is a frequency point, t k is a time point, j is an imaginary unit, π is a constant, and t is a sampling point in time domain;

[0219] Step 5.3, reconstructing the audio signal: after the inverse short-time Fourier transform is completed, the final time-domain audio signal y(t) is obtained by synthesizing each overlapping part;

[0220] Step 5.4, outputting the final audio signal: the final time-domain signal y(t) is output to an audio player for playing and saving;

[0221] Step S6, post-processing the restored time-domain signal;

[0222] Step 6.1, calculating the short-time power spectrum of the time-domain audio signal: before adaptive filtering, the short-time power spectrum of the time-domain signal y(t) restored in step S5 is calculated to analyze the frequency distribution and energy characteristics of the signal, and the short-time power spectrum is defined as follows:

[0223]

[0224] where P y (f m ,t n ) is the short-time power spectrum at the t n th frame and the f m th frequency point, M is the length of the window function, w k is a windowing function, f m is a frequency index, t n is a time frame index, j is an imaginary unit, π is a constant, and t is a sampling point in time domain;

[0225] Step 6.2, estimating the noise power spectrum: assuming that the noise signal n(t) obeys a stationary Gaussian distribution, the power spectrum is estimated by a time-averaging method:

[0226]

[0227] where is the estimated noise power spectrum, λ is a smoothing factor, and P y (f m ,t n ) is the short-time power spectrum;

[0228] Step 6.3, calculating the adaptive filtering gain: after obtaining the short-time power spectrum P y (f m ,t n ) of the audio signal and the estimated noise power spectrum Wiener filter gain G(f m ,t n ) is calculated to perform the noise reduction process:

[0229]

[0230] wherein G(f m ,t n ) is the Wiener filter gain;

[0231] γ is a gain lower limit to prevent the filter from excessively suppressing the audio signal, P y (f m ,t n ) is a short-time power spectrum, is a noise power spectrum;

[0232] Step 6.4, calculating the enhanced audio signal: using the calculated Wiener filter gain G(f m ,t n ), the audio signal y(t) outputted in step S5 is enhanced, and the enhanced signal spectrum is calculated as follows:

[0233]

[0234] wherein Y is the enhanced signal spectrum, and G(f m ,t n ) is the calculated Wiener filter gain;

[0235] The enhanced signal spectrum is then converted back to a time-domain signal through inverse Fourier transform:

[0236]

[0237] wherein y is the enhanced time-domain audio signal;

[0238] Step 6.5, outputting the enhanced audio signal: the Wiener filter enhanced time-domain signal is outputted as the output of step S6, and is used for subsequent playing, storage and further processing:

[0239]

[0240] wherein y enhanced (t) is the final enhanced audio signal;

[0241] Step S7, outputting the enhanced audio signal.

[0242] The application adopts a method of adaptive multi-level feature extraction and real-time neural network noise reduction to realize fast enhancement of input speech signals. The core technologies include adaptive window optimization of short-time Fourier transform, denoising of variational autoencoder, and noise reduction of diffusion model inverse process.

[0243] First, the input speech signal is subjected to short-time Fourier transform to extract frequency domain features, while the window function is dynamically adjusted to optimize time-frequency resolution. Then, the variational autoencoder is used for denoising to extract low-dimensional latent speech features, and the diffusion model inverse process is combined to further eliminate complex noise, thereby realizing fast and accurate speech enhancement.

[0244] The application can complete real-time speech enhancement within 20ms to meet the demand for low-latency interaction. Compared with the traditional method based on fixed filters, the application can still maintain clear speech in a complex noise environment and will not cause speech distortion due to excessive filtering.

[0245] The traditional method mainly relies on fixed filters or noise reduction algorithms based on time-frequency domain masking, and the response to sudden noise is slow, resulting in unstable speech enhancement effect. In addition, the deep learning method can improve the enhancement effect, but requires a large amount of computing resources, causing high speech processing delay and failing to meet the demand for real-time application.

[0246] The application adopts a personalized audio enhancement method based on user hearing sensation feature modeling. First, the long short-term memory network + self-attention mechanism is used to analyze the user's historical listening data to establish a personalized audio enhancement model. Second, adaptive equalization technology is used to dynamically adjust the gain for different frequency bands to match the user's hearing habits. In addition, a filter with learnable parameters is used to realize accurate personalized sound effect enhancement.

[0247] The application can dynamically adjust the audio output according to the hearing preference of different users to ensure that the user can obtain the optimal hearing experience in different devices and different environments. Compared with the traditional preset adaptive equalization mode, the method can automatically adapt to the user's demand to avoid the limitations of fixed mode sound quality.

[0248] Most traditional sound equipment adopts a fixed adaptive equalization mode and lacks the ability of personalized adaptive adjustment. In addition, some personalized audio optimization schemes based on machine learning have high training costs and limited generalization ability, resulting in unstable sound quality performance in different devices and environments.

[0249] The application adopts a multi-channel spatial perception + deep learning end-to-end speech separation technology. First, the multi-channel input is obtained by using a microphone array, and the sound source direction feature is extracted by spatial filtering. Then, the time-space attention network is combined to realize accurate speech separation. In addition, an end-to-end deep learning model is used to optimize the sound field information to make the target speech have natural spatial hierarchy.

[0250] The application can realize accurate voice separation in a multi-person voice environment, optimize sound field direction sense, and ensure accurate sound source positioning. Compared with the traditional independent sound source separation + sound field processing serial scheme, the method can avoid sound source aliasing problem, and improve the authenticity of sound quality and spatial perception.

[0251] Existing voice separation methods mostly rely on traditional blind source separation or single-channel-based deep learning models, and in a complex sound field environment, problems such as sound source aliasing and lack of direction sense are prone to occur. In addition, traditional sound field optimization relies on fixed parameters and cannot be adaptively adjusted according to the space environment, resulting in unrealistic spatial hierarchy.

[0252] The application adopts an audio compression decoding scheme combining variational autoencoder + quantization neural network. First, the variational autoencoder is used to extract low-dimensional latent features for efficient compression of audio signals while retaining key information. Then, the quantization neural network is combined for quantization processing to convert high-dimensional features into low-bit rate coding to reduce storage and transmission costs. When decoding, a deep generation network is used to restore the audio to ensure high-fidelity sound quality under low-bit rate conditions.

[0253] The application can maintain near-lossless sound quality at a bandwidth as low as 32kbps, significantly reducing data transmission requirements. Compared with traditional linear predictive coding, the audio compression ratio of the method is higher, and the sound quality loss is low, making it particularly suitable for low-bandwidth network environments.

[0254] Traditional audio encoding methods rely on fixed parameter encoding, and in low-bit rate conditions, high-frequency details are severely lost, which cannot guarantee sound quality. AI-based neural network audio encoding schemes can improve sound quality, but have high computational complexity and cannot efficiently run in embedded devices or low-power scenarios.

[0255] The audio signal processing method of the sound equipment based on artificial intelligence adopts model pruning and quantization technology to reduce the consumption of computing resources, wherein the pruning process is realized by optimizing the following objective function:

[0256] Wherein, is the optimized weight matrix, W is the weight of the original neural network, is the weight retained after pruning, Q is the quantization step, is the optimization solving process,

[0257] is the square error between the original weight W and the quantized weight .

[0258] Model pruning and quantization greatly reduce the amount of calculation, enabling AI audio processing to run on low-power devices;

[0259] By optimizing the model structure, the audio processing delay is reduced to 10-20 ms, meeting the real-time interaction requirement;

[0260] After pruning and quantization, the AI model can still guarantee high fidelity, avoiding the loss of sound quality caused by traditional compression methods;

[0261] The optimized audio AI algorithm can run on embedded chips, DSPs and mobile terminals, improving the compatibility of the system; compared with the full-precision deep learning model, the optimized scheme can reduce energy consumption by 30%-50%, and is suitable for low-power scenarios such as smart home and wireless earphones.

[0262] A terminal device includes a memory, a processor, and a computer program stored in the memory, and the computer program is executed by the processor to implement an audio signal processing method for a sound device based on artificial intelligence.

[0263] A computer-readable storage medium stores a computer program, and the computer program is executed in a computing device to implement an audio signal processing method for a sound device based on artificial intelligence.

[0264] The terminal device provided by the application integrates a memory, a processor, and a pre-installed AI audio signal processing program, realizes intelligent audio processing on the terminal side, and can complete real-time processing of audio signals locally without relying on the cloud, thereby reducing network transmission delay and improving system response speed.

[0265] The computer-readable storage medium provided by the application is used for various computing devices, and the user needs to load the storage medium to complete the deployment and running of the AI audio signal processing program.

[0266] Although embodiments of the application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the application, and the scope of the application is defined by the appended claims and their equivalents.

Claims

1. An audio signal processing method for audio equipment based on artificial intelligence, characterized in that, Comprising the following steps: Step S1, obtaining an audio signal, performing short-time Fourier transform on the audio signal, and converting it into a time-frequency domain representation; Step S2, normalizing the transformed audio signal; Step S3, inputting the normalized audio signal into a specially optimized variational autoencoder to extract a low-dimensional latent feature vector, wherein the training process of the variational autoencoder introduces manifold regularization loss to constrain the time-domain continuity of the extracted feature vector in the low-dimensional space; The step S3 further comprises: Step 3.1, Building the Variational Autoencoder: The variational autoencoder encodes the input data into a low-dimensional representation through a probabilistic model, first building the encoder network mapping the normalized spectral signal to a distribution of latent variables Z; The encoder is composed of multiple groups of convolutional layers, and the mean and variance of the latent variable are calculated: where μ Z (t) is the mean of the latent variable Z, is the variance of the latent variable Z, f μ and f σ is a neural network mapping function, and φ is a parameter, is the normalized complex frequency domain signal, f is the frequency index, and t is the time frame index. The latent variable Z is sampled using the reparameterization trick: Z(t) = μ Z (t) + σ Z (t) · β, Among them, Z(t) a ) represents the latent variables obtained from sampling, and β is the noise term of the standard normal distribution; Step 3.2, calculate the variational autoencoder objective function: the objective function of the variational autoencoder is composed of reconstruction error and KL divergence term: wherein, is the decoder-generated spectral signal, L VAE is the total loss function of the variational autoencoder; is the latent distribution KL divergence from the standard normal distribution p(Z), λ is a regularization coefficient; The KL divergence is further expanded as: where D KL is the KL divergence loss, D is the latent space dimension, and are the mean and variance of the latent variable in dimension d; Step 3.3, low-dimensional feature space regularization: introduce manifold regularization method to constrain the continuity of the features learned by the encoder in the low-dimensional space, adopt local neighborhood smoothing loss: where N(t) is the neighborhood of time frame t, w t′ is the neighborhood weighting coefficient, L MR is the manifold regularization loss; Finally, the optimization objective function of the variational autoencoder becomes: L total = L VAE + γL MR , where L total is the final optimization objective, and γ is a hyperparameter that controls the regularization strength of the control manifold. Step 3.

4. Recovering the preliminary denoised spectrum by the variational autoencoder decoder: using the decoder network Re-mapping the latent variable Z back to the time-frequency spectrum space, recovering the preliminary denoised audio signal: where g θ is a decoder network trained by parameters θ, is a denoised audio spectrum generated by the variational autoencoder; The restored spectral signal is passed to step S4 for denoising processing of the diffusion model to further remove residual noise and optimize sound quality; Step S4, input the low-dimensional latent feature vector with time-domain continuity into the diffusion model, and use the inverse diffusion process to remove noise; Step S5, inverse short-time Fourier transform is performed on the denoised audio signal to restore the time-domain signal; Step S6, apply an adaptive Wiener filter to the restored time-domain signal for post-processing to suppress residual noise; The step S6 further comprises: Step 6.1, calculate the short-time power spectrum of the time-domain audio signal: before adaptive filtering, the short-time power spectrum of the time-domain signal y(t) restored in step S5 is calculated to analyze the frequency distribution and energy characteristics of the signal, and the short-time power spectrum is defined as follows: Among them, P y (f m ,t n ) is the t-th n Frame, f m The short-time power spectrum at a frequency point, where M is the length of the window function, w k For windowing functions, f m For frequency index, t n Here, j is the time frame index, j is the imaginary unit, π is a constant, and t is the sampling point in the time domain. Step 6.2, estimate the noise power spectrum: assuming that the noise signal n(t) follows a stationary Gaussian distribution, the power spectrum is estimated by time averaging method: wherein is the estimated noise power spectrum, λ is a smoothing factor, P y (f m , t n ) is the short-time power spectrum; Step 6.3, calculate the adaptive filter gain: after obtaining the short-time power spectrum P of the audio signal. y (f m ,t n and estimated noise power spectrum Then, calculate the Wiener filter gain G(f) m ,t n To perform noise reduction processing: where G(f m ,t n ) is the Wiener filter gain, γ is a lower gain limit to prevent the filter from excessively suppressing the audio signal, P y (f m ,t n ) is the short-time power spectrum, is the noise power spectrum; Step 6.4, computation of the enhanced audio signal: The audio signal y(t) output from step S5 is enhanced using the computed Wiener filter gain G(f m ,t n ), and the enhanced signal spectrum is computed as follows: wherein G(f) is the enhanced signal spectrum, G(f m t n ) is the computed Wiener filter gain; Then convert the enhanced spectrum back to the time-domain signal through inverse Fourier transform: wherein, is the enhanced time-domain audio signal; Step 6.5, output enhanced audio signal: time domain signal enhanced by Wiener filtering As output of step S6, and for subsequent playback, storage and further processing: wherein y enhanced (t) is the final enhanced audio signal; Step S7, output the enhanced audio signal.

2. The artificial intelligence-based audio device audio signal processing method of claim 1, wherein, The step S1 further comprises: Step 1.1, Collecting the original audio signal: Obtain continuous audio signals through a microphone array and a single-channel microphone, and the discrete signals after sampling are represented as: x[n] = x(nT s ), where T is the sampling period, x[n] is the sampled discrete audio signal, and n is the discrete time index. s where T is the sampling period, x[n] is the sampled discrete audio signal, and n is the discrete time index. Step 1.2, Compute the spectrum signal: Frame the signal x[n] with a frame length N using short-time Fourier transform, and each frame signal is denoted as: x m [k] = x[k + mR], 0 ≤ k < N, where x m [k] is the signal sample in the mth frame, k is the sampling point index within the frame, m is the frame index, R is the frame shift length, and the short-time Fourier transform is calculated after windowing w[k] of the respective frame signals: where X m [f] is the spectrum of the mth frame, f is the frequency index, j is the imaginary unit, N is the frame length, e -j2πfk / N is the oscillation in the frequency domain, π is a constant, and w[k] is a window function. Step 1.3, Adjusting the window function: The window function w[k] is optimized according to the statistical properties of the audio signal. First, the short-time energy of the signal is computed: Short-time zero-crossing rate of the post-computed signal: where sgn is the sign function, E m is the short-term energy of the mth frame, Z m is the short-term zero-crossing rate of the mth frame, in combination with E m and Z m determines the parameters of the window function, such as: where a is a dynamically adjusted parameter, a is dynamically adjusted according to E m and the statistical properties of Z m .

3. The method of claim 1, wherein the method further comprises: The step S2 further comprises: Step 2.1, computing the magnitude spectrum of the input audio signal: The short-time Fourier transform of the audio signal obtained in step S1 is denoted as: X(f, t) = A(f, t)e jθ(f,t) Where X(f,t) is a frequency domain complex signal, f is a frequency index, t is a time frame index, A(f,t) is an amplitude spectrum, θ(f,t) is phase information, and j is an imaginary unit; On this basis, the amplitude spectrum A(f,t) is obtained by calculating the modulus of the complex signal X(f,t): A(f,t) = |X(f,t)|, Step 2.2, Compute global mean and standard deviation: Compute the mean and standard deviation of the amplitude spectrum for all time frames: where T is the total number of time frames, F is the total number of frequency points, μ A is the global mean amplitude of the overall audio signal, σ A is the global standard deviation of the overall audio signal; Step 2.3, normalize the amplitude spectrum: normalize the amplitude spectrum: wherein is the normalized amplitude spectrum, and ε is a numerical stability parameter. Step 2.4, amplitude limiting: limit the amplitude of the normalized amplitude spectrum to prevent extreme values from affecting the stability of the model: wherein is the amplitude spectrum after clipping, C max is the clipping threshold; Step 2.5, Normalized spectrum recovery: Based on the normalized amplitude spectrum, the original phase information θ(f, t) is kept unchanged, and the normalized frequency domain signal is recovered: a ) not changed, the normalized frequency domain signal is recovered: wherein is the normalized complex frequency domain signal; Step 2.6, output normalized spectrum signal: the normalized signal is passed to step S3 for variational autoencoder feature extraction.

4. The artificial intelligence-based audio device audio signal processing method of claim 1, wherein, The step S4 further comprises: Step 4.1, construct the forward diffusion process: the forward process of the diffusion model gradually adds noise to make the input signal tend to a standard Gaussian distribution; Given the initially denoised spectral signal The forward diffusion process is defined as follows: where Y k is the noise version signal generated at the kth step, Y k-1 is the noise version signal generated at the k-1th step, a k is the attenuation coefficient, I is the identity matrix, and k is the diffusion time step. By repeating the application of the forward diffusion process, a completely noisy signal Y is eventually obtained k : q(Y K |Y0) = N(Y K ; 0, I), where Y0is the initial input; Step 4.2, calculate the noise estimate of the inverse diffusion process: the purpose of the inverse diffusion process is to remove noise and gradually restore the signal to a high-quality audio signal from noise; The neural network parameterizes the inverse diffusion process using an estimated noise model ∈ θ (Y K |Y) to compute the mean and variance of the denoised signal: where μ θ (Y k k) is the estimated mean of the denoising network, is the variance in the denoising process, controlling the uncertainty; Estimate the mean μ θ (Y k k) is calculated by the following equation: where ∈ θ (Y k k) is the noise component estimated by the neural network. Step 4.3, Compute the loss function of the diffusion model: The optimization goal of the diffusion model is to minimize the mean squared error between the real noise ∈ and the estimated noise ∈ θ (Y k ,k) where L DM is the loss function of the diffusion model, E is the expectation calculation for all samples, and ∈ is the real noise. By minimizing L DM , the neural network is trained to estimate the noise component correctly at different time steps k, achieving denoising; Step 4.4, variable step size inverse diffusion process optimization: Adopting variable step size diffusion strategy, dynamically adjusting step size a k : Wherein, τ is a dynamic factor to control the step length change, K is the total number of steps of the diffusion process; Step 4.5, output de-noised audio signal: after diffusion model de-noising, the optimized audio spectrum is finally obtained where Y1 is the noise version signal generated in step 1. The de-noised spectrum signal is transmitted to step S5 for inverse short-time Fourier transform to restore the time-domain audio signal for subsequent audio enhancement and playing.

5. The artificial intelligence-based audio device audio signal processing method of claim 1, wherein, The step S5 further comprises: Step 5.1, setting inverse short-time Fourier transform parameters: before inverse short-time Fourier transform, necessary inverse short-time Fourier transform parameters are set; Step 5.2, inverse short-time Fourier transform calculation: after setting the parameters of inverse short-time Fourier transform, inverse transform is performed, and the frequency domain signal is restored to the time domain signal y(t) using the following formula: where y(t) is the recovered time-domain audio signal, is the recovered de-noised frequency-domain signal, w(t k ) is a window function, f k is a frequency point, t k is a time point, j is the imaginary unit, π is a constant, and t is a time-domain sample point; Step 5.3, reconstructing the audio signal: after the inverse short-time Fourier transform is completed, the final time-domain audio signal y(t) is obtained by synthesizing each overlapping part; Step 5.4, outputting the final audio signal: the final time-domain signal y(t) is output to the audio player for playing and saving.

6. The artificial intelligence-based audio device audio signal processing method of claim 1, wherein, The artificial intelligence-based sound equipment audio signal processing method adopts model pruning and quantization technology to reduce the consumption of computing resources, wherein the pruning process is realized by optimizing the following objective function: wherein, W is the weight of the original neural network, Q is the quantization step, is the optimization process, is the square error between the original weight W and the quantized weight Wq.

7. A terminal device, characterized by comprising: The computer program is stored in the memory and is executed by the processor to implement the artificial intelligence-based audio signal processing method of the audio device according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer program is stored in the memory and is executed by the processor to implement the artificial intelligence-based audio signal processing method of the audio device according to any one of claims 1 to 6.

Images