Audio noise reduction method and device

Abstract

The invention discloses an audio noise reduction method and device, and belongs to the technical field of audio processing. The method comprises the following steps: performing equivalent rectangular bandwidth conversion processing on a first audio signal to obtain a first audio feature; the first audio feature is input into an audio noise reduction model, the audio noise reduction model comprises an encoder, a processor and a decoder, the first audio feature is encoded through the encoder, convolution operation is carried out on an encoded audio feature vector through the processor, the convolution feature vector after convolution operation is decoded through the decoder, and the first audio feature is input into the audio noise reduction model; obtaining a second audio feature; the processor comprises an enhanced time sequence convolutional network block, and the enhanced time sequence convolutional network block comprises a causal expansion convolution branch; and performing inverse equivalent rectangular bandwidth conversion processing on the second audio feature to obtain a second audio signal.

Description

Technical Field

[0001] This application belongs to the field of audio processing technology, specifically relating to an audio noise reduction method and apparatus. Background Technology

[0002] Audio denoising technology aims to remove noise interference from noisy audio signals while preserving the clarity and naturalness of the target speech or audio. It is widely used in voice communication, smart devices, audio and video conferencing, and medical diagnostics. Traditional audio denoising methods are based on assumptions about the statistical properties of signals and are effective against stationary noise. However, they easily lead to distortion of the target signal when faced with non-stationary noise, and it is difficult to balance denoising effectiveness with signal preservation. Therefore, deep learning methods have emerged and are widely used in audio denoising, improving performance by learning the mapping relationship between noise and clean signals. However, current deep learning audio denoising models suffer from poor denoising performance due to insufficient capture of temporal dependencies and inadequate consideration of the nonlinear characteristics of the human auditory system. Summary of the Invention

[0003] The purpose of this application is to provide an audio noise reduction method and apparatus that can improve the noise reduction effect under complex noise, reduce audio distortion, and improve noise reduction efficiency to meet the real-time requirements of voice interaction.

[0004] In a first aspect, embodiments of this application provide an audio noise reduction method, the method comprising: The first audio signal is subjected to an equivalent rectangular bandwidth transformation to obtain the first audio feature; The first audio feature is input into the audio denoising model, which includes an encoder, a processor, and a decoder. The encoder encodes the first audio feature to obtain an audio feature vector. The processor performs a convolution operation on the audio feature vector to obtain a convolutional feature vector. The decoder decodes the convolutional feature vector to obtain the second audio feature. The processor includes an enhanced temporal convolutional network block, which includes a causal dilated convolutional branch. The second audio feature is subjected to inverse equivalent rectangular bandwidth transformation to obtain the second audio signal.

[0005] Secondly, embodiments of this application provide an audio noise reduction device, the device comprising: The first processing module is used to perform equivalent rectangular bandwidth transformation on the first audio signal to obtain the first audio features; The model processing module is used to input the first audio feature into the audio denoising model. The audio denoising model includes an encoder, a processor, and a decoder. The encoder encodes the first audio feature to obtain an audio feature vector. The processor performs a convolution operation on the audio feature vector to obtain a convolutional feature vector. The decoder decodes the convolutional feature vector to obtain the second audio feature. The processor includes an enhanced temporal convolutional network block, which includes a causal dilated convolutional branch. The second processing module is used to perform inverse equivalent rectangular bandwidth transformation on the second audio feature to obtain the second audio signal.

[0006] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0007] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0008] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0009] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0010] In this embodiment, an equivalent rectangular bandwidth transform is performed on a first audio signal to obtain a first audio feature. The first audio feature is then input into an audio denoising model, which includes an encoder, a processor, and a decoder. The encoder encodes the first audio feature to obtain an audio feature vector, the processor performs a convolution operation on the audio feature vector to obtain a convolutional feature vector, and the decoder decodes the convolutional feature vector to obtain a second audio feature. The processor includes an enhanced temporal convolutional network block, which includes a causal dilated convolutional branch. The second audio feature is then subjected to an inverse equivalent rectangular bandwidth transform to obtain a second audio signal.

[0011] In this way, the equivalent rectangular bandwidth transformation, based on the characteristics of the human auditory filter bank, preserves high resolution for low-frequency signals and reduces redundancy for high-frequency signals, making the extracted first audio features more consistent with the human auditory perception mechanism. After being input into the audio denoising model, the enhanced temporal convolutional network block captures long-term dependencies in the audio signal through causal dilation convolutional branches, while maintaining temporal causality, thus increasing the temporal receptive field and reducing inference latency. In this way, the denoising effect is improved under complex noise, audio distortion is reduced, and denoising efficiency is increased to meet the real-time requirements of voice interaction. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating an audio noise reduction method provided in some embodiments of this application; Figure 2 This is a schematic diagram illustrating the working principle of an audio noise reduction method provided in some embodiments of this application; Figure 3 This is a schematic diagram of the structure of the audio noise reduction model in the audio noise reduction method provided in some embodiments of this application; Figure 4 This is a schematic diagram of the first loss function in the audio noise reduction method provided in some embodiments of this application; Figure 5 This is a schematic diagram of the structure of an audio noise reduction device provided in some embodiments of this application; Figure 6 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application; Figure 7 These are schematic diagrams of the hardware structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0015] Audio noise reduction technology aims to remove noise interference from noisy audio signals while preserving the clarity and naturalness of the target speech or audio. It is widely used in voice communication, smart devices, audio and video conferencing, and medical diagnosis. In particular, with the development of smart wearable devices, portable interactive devices such as Artificial Intelligence (AI) glasses are gradually becoming important carriers of human-computer interaction. These devices often integrate microphone arrays for voice interaction, calls, and recording; however, in real-world scenarios, they encounter complex interference from environmental noise, wind noise, electronic noise, and indoor reverberation, significantly affecting the quality of voice acquisition and user experience.

[0016] Currently, audio noise reduction technologies mainly include traditional signal processing methods, such as spectral subtraction, Wiener filtering, and Kalman filtering, as well as deep learning methods.

[0017] Traditional signal processing methods are computationally efficient but poorly adaptable to non-stationary noise, easily producing musical noise and distortion. Deep learning methods suffer from a disconnect from subjective auditory perception, limited temporal modeling and latency, and high deployment costs. Specifically, most deep learning methods use signal-level losses such as mean squared error (MSE), failing to incorporate the psychoacoustic characteristics of the human ear, easily leading to perceptual distortions such as "muffled" or "spiculated" sounds. Structures primarily based on recurrent neural networks (RNNs) such as gated recurrent units (GRUs) have limited effective receptive fields, and serial computation results in higher inference latency, hindering real-time interaction. The high number of parameters and computational complexity make stable operation difficult under the power consumption, computing power, and memory constraints of edge devices. Training data and structures are largely geared towards general call scenarios, without optimization for glasses wearing positions, motion states, and wind noise characteristics, resulting in insufficient generalization and robustness.

[0018] Based on this, embodiments of this application provide an audio noise reduction method and apparatus to solve the aforementioned problems. To facilitate understanding of the technical solutions of these embodiments, the following description will use the noise reduction processing of audio signals collected by AI glasses in a wearable device as an example. It is understood that this example is merely intended to illustrate the audio noise reduction method provided in these embodiments and is not intended to limit the scope of the audio noise reduction method.

[0019] The audio noise reduction method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0020] Figure 1 This is a schematic flowchart of the audio noise reduction method provided in this application embodiment. The audio noise reduction method may include: Step 101: Perform equivalent rectangular bandwidth transformation on the first audio signal to obtain the first audio feature.

[0021] In step 101, as Figure 2 As shown, the entire system architecture may include an audio input module 201, which may be a dual-microphone array on a wearable device for acquiring a first audio signal, which may be an audio signal to be denoised.

[0022] The first audio signal can be sampled at a sampling rate of 16kHz and then enter the preprocessing module 202. The preprocessing module 202 can perform a short-time fourier transform (STFT) on the sampled first audio signal to convert the time domain signal to the frequency domain, in preparation for the subsequent equivalent rectangular bandwidth (ERB) transform.

[0023] For example, by segmenting and windowing the time-domain signal, and then performing a Fourier transform on the signal within each window, a one-dimensional time-domain signal can be converted into a two-dimensional "time-frequency" spectrum, thereby simultaneously reflecting the frequency components and amplitude information of the signal at different time points.

[0024] The first audio signal, converted into a two-dimensional "time-frequency" spectrum, can enter the ERB transformation module 203 to perform ERB transformation processing. ERB is a frequency band division method that simulates the characteristics of human hearing. The first audio signal can be decomposed into ERB frequency bands through a 39-band filter bank to obtain the ERB spectrum, thus enabling better extraction of the first audio features related to auditory perception.

[0025] Step 102: Input the first audio feature into the audio denoising model. The audio denoising model includes an encoder, a processor, and a decoder. The encoder encodes the first audio feature to obtain an audio feature vector. The processor performs a convolution operation on the audio feature vector to obtain a convolutional feature vector. The decoder decodes the convolutional feature vector to obtain the second audio feature. The processor includes an enhanced temporal convolutional network block, which includes a causal dilated convolutional branch.

[0026] In step 102, as Figure 2 As shown, the first audio feature can be input into the audio denoising model 204. The core of the audio denoising model 204 can be an enhanced-temporal convolutional network (Enhanced-TCN).

[0027] Among them, such as Figure 3 As shown, the audio denoising model may include an encoder 301, a processor 302, and a decoder 303. The processor 302 may include an Enhanced-TCN, which may include parallel causal dilated convolutional branches. This can capture long-term dependencies in the audio signal while maintaining temporal causality, meaning the output depends only on the current and past inputs, which is crucial for real-time audio processing. The causal dilated convolutional branches can also expand the receptive field without increasing the number of parameters, thus improving model performance.

[0028] The first audio feature can be encoded by encoder 301 to obtain an audio feature vector. The audio feature vector is then convolved by processor 302 to obtain a convolutional feature vector. Finally, the convolutional feature vector is decoded by decoder 303 to obtain the second audio feature. This second audio feature can correspond to the enhanced ERB spectrum after ERB spectrum enhancement.

[0029] Step 103: Perform inverse equivalent rectangular bandwidth transformation on the second audio feature to obtain the second audio signal.

[0030] In step 103, as Figure 2 As shown, the second audio feature can enter the inverse ERB transform module 205 to perform inverse ERB transform processing on the second audio feature, and then enter the post-processing module 206 after obtaining the frequency domain signal of the spectrum reconstruction. After the post-processing module 206 performs inverse fast fourier transform (IFFT) processing on the second audio feature, the frequency domain signal is inversely transformed to restore the time domain signal. The time domain signal is then concatenated to obtain the second audio signal, and then the audio output module 207 can output the second audio signal. The second audio signal can be a noise-reduced enhanced audio signal.

[0031] In this embodiment, the audio denoising method performs an equivalent rectangular bandwidth transformation on a first audio signal to obtain a first audio feature; the first audio feature is input into an audio denoising model, which includes an encoder, a processor, and a decoder. The encoder encodes the first audio feature to obtain an audio feature vector; the processor performs a convolution operation on the audio feature vector to obtain a convolutional feature vector; and the decoder decodes the convolutional feature vector to obtain a second audio feature. The processor includes an enhanced temporal convolutional network block, which includes a causal dilated convolutional branch. The second audio feature is then subjected to an inverse equivalent rectangular bandwidth transformation to obtain a second audio signal.

[0032] In this way, the equivalent rectangular bandwidth transformation, based on the characteristics of the human auditory filter bank, preserves high resolution for low-frequency signals and reduces redundancy for high-frequency signals, making the extracted first audio features more consistent with the human auditory perception mechanism. After being input into the audio denoising model, the enhanced temporal convolutional network block captures long-term dependencies in the audio signal through causal dilation convolutional branches, while maintaining temporal causality, thus increasing the temporal receptive field and reducing inference latency. In this way, the denoising effect is improved under complex noise, audio distortion is reduced, and denoising efficiency is increased to meet the real-time requirements of voice interaction.

[0033] In some embodiments, the number of enhanced temporal convolutional network blocks is B, and each enhanced temporal convolutional network block includes K causal dilated convolutional branches; The output value of the b-th enhanced temporal convolutional network block is obtained by fusing the output values ​​of the K causal dilated convolutional branches of the b-th enhanced temporal convolutional network block with the output value of the (b-1)-th enhanced temporal convolutional network block; B and K are integers greater than 1, and b is a positive integer less than or equal to B.

[0034] In this embodiment, as Figure 3 As shown, the processor 302 can be composed of B Enhanced-TCN blocks cascaded together. Each Enhanced-TCN block contains K parallel causal dilated convolution branches, which can capture dependencies at different time scales simultaneously.

[0035] No. b Output value of each Enhanced-TCN block The output values ​​of the K causal dilated convolutional branches of the b-th enhanced temporal convolutional network block are fused with the output value of the (b-1)-th enhanced temporal convolutional network block, as shown in formula (1): in, It is the output value of the previous Enhanced-TCN block, passed in through the residual connection. It is the output value of the k-th causal dilated convolution branch in the b-th block. It is a feature map that is pieced together along the channel dimension, connecting all branches. It is an attention module used to adaptively learn the importance of each feature and perform weighted fusion.

[0036] That is, the convolutional feature vector can be the output value of the last Enhanced-TCN block, that is, the output value of the B-th Enhanced-TCN block.

[0037] The core of each branch is the causal dilated convolution, whose operation in the time dimension t can be represented as shown in Equation (2): in, These are the weights of the convolution kernel, and L is the kernel size. The expansion rate of the k-th branch. This design allows the network's receptive field to grow exponentially with the number of layers B, enabling efficient modeling of speech context lasting several seconds, while its parallel computing characteristics significantly reduce inference latency.

[0038] In this way, multiple Enhanced-TCN blocks allow the model to learn more complex feature levels, improving noise reduction performance. Furthermore, Enhanced-TCN blocks effectively capture long temporal dependencies and multi-scale features through multi-branch causal dilated convolution and cross-block fusion, significantly increasing the temporal receptive field while significantly reducing inference latency.

[0039] In some embodiments, before performing an equivalent rectangular bandwidth transformation on the first audio signal to obtain the first audio features, the method may further include: Acquire the raw audio signal; The original audio signal is subjected to inverse filtering and compensation processing to obtain the first audio signal.

[0040] In this embodiment, the raw audio signal collected by the dual-microphone array can be acquired. It is understood that the wearing position of the AI ​​glasses introduces specific spectral coloration, which may lead to spectral distortion.

[0041] Based on this, a pre-measured database of head-related transfer functions can be used. The audio spectrum of the raw audio signal acquired by the dual-microphone array. Inverse filtering compensation is performed to restore its original spectral characteristics.

[0042] The inverse filtering compensation process can be shown in formula (3): in, It is an HRTF inverse filter corresponding to a specific wearing position, used to correct spectral distortion.

[0043] In this way, the original audio signal may be distorted in frequency response due to the hardware filtering characteristics of the acquisition device. Inverse filtering compensation corrects the amplitude and phase frequency distortions introduced by the device, making the first audio signal input to subsequent processing closer to the characteristics of the real sound source, improving the accuracy of ERB transform features, and indirectly improving the input quality of the audio noise reduction model and the final audio noise reduction effect.

[0044] In some embodiments, before inputting the first audio feature into the audio noise reduction model, the method may further include: Acquire sensing information collected by multimodal sensors; Determine scene information based on perceived information; Based on scene information, an audio noise reduction model corresponding to the scene information is determined; the model parameters of the audio noise reduction model are different for different scene information.

[0045] In this embodiment, in order to cope with the complex and ever-changing acoustic environment in the real world, an adaptive noise reduction mechanism can be used to enable the AI ​​glasses to intelligently adjust the noise reduction strategy.

[0046] It can acquire perception information collected by multimodal sensors, which may include data from the inertial measurement unit built into the AI ​​glasses. and barometer data p Etc. Inertial measurement unit data, among others. It can be used to detect motion status and barometer data. p It can be used to help determine altitude and whether it is indoors or outdoors.

[0047] Based on perceived information, the user's current scene can be determined. For example, multiple scenes can be pre-defined, each associated with corresponding reference perceived information. The collected perceived information can be matched with the reference perceived information for multiple scenes to determine the scene information to which the perceived information belongs.

[0048] In some embodiments, determining scene information based on perception information may include: The perceived information is input into the scene classifier, which then identifies the scene information.

[0049] In this embodiment, a lightweight scene classifier can be used. Based on the perceived information, it can identify the user's scene information in real time. Scene classifier The expression for can be shown in formula (4): In this way, using a scene classifier to automatically identify the environment type and determine the scene information can simplify the process and improve efficiency.

[0050] Based on the identified scene information, the system automatically executes corresponding adaptive noise reduction strategies. For example, in a "cycling" scene, a specially optimized wind noise suppression module is activated; in a "street" scene, background ambient sound is appropriately preserved to ensure the wearer's safety. It's understandable that different noise reduction strategies correspond to audio noise reduction models with different parameters to achieve different audio noise reduction optimization effects.

[0051] In this way, the noise characteristics of different scenarios vary significantly. By acquiring scene information through multimodal sensors and calling model parameters adapted to the scenario, the noise reduction strategy can be dynamically optimized, avoiding the effect fluctuation of a fixed model in complex scenarios and improving the robustness of noise reduction and scene adaptability.

[0052] In some embodiments, before inputting the first audio feature into the audio noise reduction model, the method may further include: Obtain multiple training samples; Construct an initial model; the first loss function of the initial model includes a perceptual loss function, a temporal envelope loss function, and a phase consistency loss function. The perceptual loss function is used to characterize the perceptual loudness error corresponding to the masking effect and frequency sensitivity. The temporal envelope loss function is used to ensure the macroscopic dynamics and energy fluctuations of the audio signal. The phase consistency loss function is used to penalize the phase spectrum inconsistency of the audio signal before and after noise reduction. The initial model is trained using multiple training samples until the initial model is determined to have converged based on the first loss function, thus obtaining the audio noise reduction model.

[0053] In this embodiment, multiple training samples can be obtained. Each training sample may include a first audio feature sample after the noise reduction frequency signal sample has undergone ERB transformation, and a second audio signal sample corresponding to the first audio feature sample. The second audio signal sample can represent a clean audio signal.

[0054] An initial model can be constructed. This initial model, as understood, has the same architecture as the audio denoising model, including an encoder, processor, and decoder. The first audio feature sample can be input into the initial model. Based on the predicted audio signal output from the initial model and a second audio signal sample, the loss value is calculated by substituting these values ​​into the initial model's first loss function. The model parameters are then optimized based on this loss value until the loss value satisfies the model convergence condition, resulting in a trained audio denoising model. The predicted audio signal can represent the denoised audio signal.

[0055] To fundamentally address the disconnect between the optimization objective and subjective listening experience in traditional noise reduction methods, a composite loss function can be constructed. This serves as the first loss function for the initial model. The first loss function guides the training process of the neural network, ensuring that its optimization direction aligns with the perceptual characteristics of the human ear.

[0056] like Figure 4As shown, the construction process of the first loss function based on human ear perception includes three parallel loss calculation branches, thereby generating more natural and clear speech. The first loss function 400 is composed of three weighted parts: the perceptual loss function 401, the temporal envelope loss function 402, and the phase consistency loss function 403. The expression of the first loss function 400 can be shown in formula (5): in, For the first loss function, , , These represent the perceptual loss function, the temporal envelope loss function, and the phase consistency loss function, respectively. , , Hyperparameters used to balance the importance of different loss functions.

[0057] Perceptual loss function Several principles of auditory psychoacoustics are incorporated into the loss calculation. First, to simulate the nonlinear frequency response of the human cochlea, the linear spectra of the pure audio signal 4011 and the noise-reduced frequency signal 4012 are compared. The ERB transform 4013, performed using an ERB filter bank, converts the spectrum into an ERB spectrogram that better matches auditory perception. .

[0058] Secondly, the Stevens power law is applied to map physical energy to perceived loudness 4014, to simulate the logarithmic perception of sound intensity by the human ear. The expression for perceived loudness can be shown in formula (6): in, To perceive loudness; These represent the indices for time frames, linear frequencies, and ERB bands, respectively. It is the first Frame, First Signal energy in each ERB band It is the compression factor of perceived loudness, used to simulate the perceived compression effect.

[0059] The perceptual loss function can be obtained by applying frequency sensitivity weighting (4015) and masking effect weighting (4016) based on the perceived loudness (4014). In other words, the final perceptual loss function It can be used for the perceptual loudness error, which is defined as being weighted by both masking effect and frequency sensitivity, and the perceptual loss function. The expression for can be shown in formula (7): in, The perceived loudness is measured by the noise-reduced frequency signal and the pure audio signal, respectively. This is the frequency sensitivity weight. The weight is designed based on the ISO 226 equal loudness curve, giving higher weight to the 1-5kHz key speech frequency band that the human ear is most sensitive to. It is the auditory masking weight, based on the auditory masking threshold. Dynamic adjustment. When the prediction error is below this threshold, i.e., the error is masked by the loud noise and becomes inaudible, this weight is set to a smaller value, so that the model ignores these insignificant errors. These are the total number of audio time frames and the total number of ERB bands, respectively.

[0060] To further improve audio quality, two auxiliary loss functions are introduced: the time domain envelope loss function 402 and the phase consistency loss function 403.

[0061] The temporal envelope loss function 402 is used to preserve the macroscopic dynamics and energy fluctuations of the speech, preventing the denoised sound from sounding "muffled" or lacking power. Temporal envelope loss function The expression for can be shown in formula (8): in, , It is a time-domain waveform signal of pure audio signal and noise-reduced frequency signal. This represents a low-pass filter used to smoothly extract the waveform envelope. Represents the L1 norm.

[0062] The phase consistency loss function 403 is used to penalize the phase spectrum inconsistency of the audio before and after noise reduction, effectively reducing the "metallic" or "hissing" sound caused by phase distortion. The expression for can be shown in formula (9): in, It is the STFT complex spectrum of the pure audio signal and the noise-reduced frequency signal. It is a very small positive number used to prevent the denominator from being zero. This represents the square of the L2 norm, i.e., the mean square error.

[0063] In this way, perceptual loss directly optimizes the perceived loudness error of denoised audio by simulating the masking effect and frequency sensitivity of the human ear. Temporal envelope loss reflects the macroscopic energy fluctuations of the audio; this constraint ensures that the dynamic range of the denoised audio is not overly smoothed, improving naturalness. Phase consistency loss reduces phase distortion by penalizing the difference in phase spectrum before and after denoising, ensuring phase consistency and improving the auditory realism of the audio. The three factors work synergistically to optimize the model training objective from the three dimensions of perception, dynamics, and phase, significantly improving the subjective experience of denoised audio.

[0064] In some embodiments, training an initial model based on multiple training samples until the initial model is determined to have converged based on a first loss function, yielding an audio denoising model, may include: The initial model is trained based on multiple training samples until the initial model is determined to have converged based on the first loss function, at which point the first model is obtained. With the first model being the teacher model, a lightweight student model is constructed. A second loss function for the student model is generated based on the distillation loss function and the supervised loss function. The distillation loss function is used to represent the difference in feature distribution output by the teacher model and the student model in the intermediate layers of the network, and the supervised loss function is used to represent the difference in probability distribution output by the teacher model and the student model. The student model is trained by knowledge distillation based on multiple training samples and a second loss function to obtain an audio noise reduction model.

[0065] In this embodiment, in order to successfully deploy the high-performance noise reduction model on resource-constrained edge devices such as AI glasses, the trained model can be compressed and hardware co-optimized.

[0066] For example, as mentioned above, the initial model can be trained on multiple training samples until convergence is determined based on the first loss function, resulting in a first model with a large number of parameters and superior performance. This first model can then be used as a teacher model. This teacher model can be used as a basis. Build a lightweight student model This allows the model to not only learn to fit real data, but also to learn to imitate the output of the teacher's model.

[0067] Student model Feature scale of feature layer and teacher model Feature scale alignment is performed on the feature layers, and a distillation loss function is determined. The distillation loss function represents the difference in feature distribution output by the teacher model and the student model in the intermediate layers of the network after feature scale alignment.

[0068] Student model The probability distribution of the predicted values ​​of the output layer and the teacher model The probability distributions corresponding to the predicted values ​​of the output layer are aligned, and the supervised loss function is determined. The supervised loss function is used to represent the difference in probability distributions between the teacher model and the student model outputs.

[0069] A second loss function can be generated for the student model based on the distillation loss function and the supervised loss function. Second loss function The expression for can be shown in formula (10): in, This is the final output value of the student model. It represents the feature distribution output by the teacher model and student model in the intermediate layer of the network. It is the KL divergence, used to measure the difference between the output distributions of the student model and the teacher model. It is the weight that balances the standard loss and the distillation loss.

[0070] The student model can be trained using knowledge distillation based on multiple training samples and a second loss function to obtain an audio denoising model. Specifically, the loss value is calculated based on the second loss function, and the model parameters of the student model are optimized based on this loss value until the loss value meets the model convergence condition, resulting in a well-trained audio denoising model.

[0071] In this way, the first model, which usually has a large number of parameters and excellent performance but is difficult to deploy, can be used as the teacher model. The lightweight student model transfers the "hidden knowledge" of the teacher model through knowledge distillation. While maintaining performance close to that of the teacher model, the number of parameters and computation are greatly reduced. This solves the contradiction between high-performance models and low-computing-power devices and improves the practicality of audio noise reduction models.

[0072] In some embodiments, the student model is trained by knowledge distillation based on multiple training samples and a second loss function to obtain an audio noise reduction model, which may include: The student model is trained by knowledge distillation based on multiple training samples and the second loss function to obtain the second model. Determine the importance score for each convolutional channel in the second model; The convolutional channels in the second model with importance scores below the importance threshold are removed to obtain the audio noise reduction model.

[0073] In this embodiment, the trained learning model can be further subjected to structured pruning and mixed precision quantization. For example, as mentioned above, the student model can be trained using knowledge distillation based on multiple training samples and a second loss function to obtain a second model. That is, the second model is the trained student model.

[0074] Redundant network structures can be further removed from the lightweight second model. This can be achieved by evaluating the importance score of each convolutional channel. To perform pruning. The importance score is... The calculation formula is shown in formula (11): in, It consists of all the weight parameters associated with channel c in the convolution kernel.

[0075] Channels with importance scores below a preset importance threshold, along with their corresponding weights, can be removed to obtain an audio noise reduction model. This significantly reduces the number of model parameters and computational cost without altering the intensive computational pattern.

[0076] In this way, redundant channels can be removed by evaluating the importance score of convolutional channels, thereby reducing model parameters and computational load while keeping performance loss under control and improving inference speed.

[0077] In some embodiments, convolutional channels with importance scores less than an importance threshold in the second model are removed to obtain an audio noise reduction model, which may include: The convolutional channels in the second model with importance scores lower than the importance threshold are removed to obtain the third model; The third model is subjected to mixed precision quantization to obtain the audio noise reduction model.

[0078] In this embodiment, channels with importance scores below a preset importance threshold and their corresponding weights in the second model can be removed to obtain a third model. The third model can also be subjected to mixed-precision quantization to obtain an audio noise reduction model. For example, the weights of the third model can be... The activation value is converted from a 32-bit floating-point (FP32) value to a low-bit-width integer to reduce memory footprint and take advantage of the hardware’s integer arithmetic acceleration capabilities.

[0079] Correspondingly, the network output values ​​of each layer of the third model are quantized, and the different precision combinations of the quantized third model are tested. The best quantization precision combination is selected from the test accuracy results.

[0080] For example, in this embodiment, the optimal combination of quantization precision can be INT8 quantization for computationally intensive convolutional layers and FP16 precision reserved for quantization-sensitive input / output layers. The expression for mixed-precision quantization is shown in Equation (12): in, These are the scale factor and zero point required for quantization, respectively. This indicates rounding to the nearest integer. The result is clipped to the range of INT8 representation.

[0081] In this way, by quantizing model parameters from high precision to low precision, memory usage and computation time are reduced, while ensuring that the numerical accuracy meets the noise reduction requirements. The combination of structured pruning and mixed-precision quantization achieves "ultra-lightweight" model, further meeting the real-time noise reduction requirements of edge devices and balancing performance and deployment efficiency.

[0082] The audio noise reduction method provided in this application can be implemented by an audio noise reduction device. This application uses an audio noise reduction device to perform the audio noise reduction method as an example to illustrate the audio noise reduction device provided in this application.

[0083] like Figure 5 As shown, the audio noise reduction device 500 provided in this application embodiment may include: The first processing module 501 is used to perform equivalent rectangular bandwidth transformation on the first audio signal to obtain the first audio features; The model processing module 502 is used to input the first audio feature into the audio denoising model. The audio denoising model includes an encoder, a processor, and a decoder. The encoder encodes the first audio feature to obtain an audio feature vector. The processor performs a convolution operation on the audio feature vector to obtain a convolutional feature vector. The decoder decodes the convolutional feature vector to obtain the second audio feature. The processor includes an enhanced temporal convolutional network block, which includes a causal dilated convolutional branch. The second processing module 503 is used to perform inverse equivalent rectangular bandwidth transformation on the second audio features to obtain the second audio signal.

[0084] In this way, the equivalent rectangular bandwidth transformation, based on the characteristics of the human auditory filter bank, preserves high resolution for low-frequency signals and reduces redundancy for high-frequency signals, making the extracted first audio features more consistent with the human auditory perception mechanism. After being input into the audio denoising model, the enhanced temporal convolutional network block captures long-term dependencies in the audio signal through causal dilation convolutional branches, while maintaining temporal causality, thus increasing the temporal receptive field and reducing inference latency. In this way, the denoising effect is improved under complex noise, audio distortion is reduced, and denoising efficiency is increased to meet the real-time requirements of voice interaction.

[0085] In some embodiments, the number of enhanced temporal convolutional network blocks is B, and each enhanced temporal convolutional network block includes K causal dilated convolutional branches; The output value of the b-th enhanced temporal convolutional network block is obtained by fusing the output values ​​of the K causal dilated convolutional branches of the b-th enhanced temporal convolutional network block with the output value of the (b-1)-th enhanced temporal convolutional network block; B and K are integers greater than 1, and b is a positive integer less than or equal to B.

[0086] In this way, multiple Enhanced-TCN blocks allow the model to learn more complex feature levels, improving noise reduction performance. Furthermore, Enhanced-TCN blocks effectively capture long temporal dependencies and multi-scale features through multi-branch causal dilated convolution and cross-block fusion, significantly increasing the temporal receptive field while significantly reducing inference latency.

[0087] In some embodiments, the audio noise reduction device 500 may further include: The first acquisition module is used to acquire the raw audio signal; The third processing module is used to perform inverse filtering and compensation processing on the original audio signal to obtain the first audio signal.

[0088] In this way, the original audio signal may be distorted in frequency response due to the hardware filtering characteristics of the acquisition device. Inverse filtering compensation corrects the amplitude and phase frequency distortions introduced by the device, making the first audio signal input to subsequent processing closer to the characteristics of the real sound source, improving the accuracy of ERB transform features, and indirectly improving the input quality of the audio noise reduction model and the final audio noise reduction effect.

[0089] In some embodiments, the audio noise reduction device 500 may further include: The second acquisition module is used to acquire the sensing information collected by the multimodal sensor; The first determining module is used to determine scene information based on perceived information; The second determining module is used to determine the audio noise reduction model corresponding to the scene information based on the scene information; wherein, the model parameters of the audio noise reduction model corresponding to different scene information are different.

[0090] In this way, the noise characteristics of different scenarios vary significantly. By acquiring scene information through multimodal sensors and calling model parameters adapted to the scenario, the noise reduction strategy can be dynamically optimized, avoiding the effect fluctuation of a fixed model in complex scenarios and improving the robustness of noise reduction and scene adaptability.

[0091] In some embodiments, determining scene information based on perception information may include: The perceived information is input into the scene classifier, which then identifies the scene information.

[0092] In this way, using a scene classifier to automatically identify the environment type and determine the scene information can simplify the process and improve efficiency.

[0093] In some embodiments, the audio noise reduction device 500 may further include a model training module for: Obtain multiple training samples; Construct an initial model; the first loss function of the initial model includes a perceptual loss function, a temporal envelope loss function, and a phase consistency loss function. The perceptual loss function is used to characterize the perceptual loudness error corresponding to the masking effect and frequency sensitivity. The temporal envelope loss function is used to ensure the macroscopic dynamics and energy fluctuations of the audio signal. The phase consistency loss function is used to penalize the phase spectrum inconsistency of the audio signal before and after noise reduction. The initial model is trained using multiple training samples until the initial model is determined to have converged based on the first loss function, thus obtaining the audio noise reduction model.

[0094] In this way, perceptual loss directly optimizes the perceived loudness error of denoised audio by simulating the masking effect and frequency sensitivity of the human ear. Temporal envelope loss reflects the macroscopic energy fluctuations of the audio; this constraint ensures that the dynamic range of the denoised audio is not overly smoothed, improving naturalness. Phase consistency loss reduces phase distortion by penalizing the difference in phase spectrum before and after denoising, ensuring phase consistency and improving the auditory realism of the audio. The three factors work synergistically to optimize the model training objective from the three dimensions of perception, dynamics, and phase, significantly improving the subjective experience of denoised audio.

[0095] In some embodiments, the model training module can also be used for: The initial model is trained based on multiple training samples until the initial model is determined to have converged based on the first loss function, at which point the first model is obtained. With the first model being the teacher model, a lightweight student model is constructed. A second loss function for the student model is generated based on the distillation loss function and the supervised loss function. The distillation loss function is used to represent the difference in feature distribution output by the teacher model and the student model in the intermediate layers of the network, and the supervised loss function is used to represent the difference in probability distribution output by the teacher model and the student model. The student model is trained by knowledge distillation based on multiple training samples and a second loss function to obtain an audio noise reduction model.

[0096] In this way, the first model, which usually has a large number of parameters and excellent performance but is difficult to deploy, can be used as the teacher model. The lightweight student model transfers the "hidden knowledge" of the teacher model through knowledge distillation. While maintaining performance close to that of the teacher model, the number of parameters and computation are greatly reduced. This solves the contradiction between high-performance models and low-computing-power devices and improves the practicality of audio noise reduction models.

[0097] In some embodiments, the model training module can also be used for: The student model is trained by knowledge distillation based on multiple training samples and the second loss function to obtain the second model. Determine the importance score for each convolutional channel in the second model; The convolutional channels in the second model with importance scores below the importance threshold are removed to obtain the audio noise reduction model.

[0098] In this way, redundant channels can be removed by evaluating the importance score of convolutional channels, thereby reducing model parameters and computational load while keeping performance loss under control and improving inference speed.

[0099] In some embodiments, the model training module can also be used for: The convolutional channels in the second model with importance scores lower than the importance threshold are removed to obtain the third model; The third model is subjected to mixed precision quantization to obtain the audio noise reduction model.

[0100] In this way, by quantizing model parameters from high precision to low precision, memory usage and computation time are reduced, while ensuring that the numerical accuracy meets the noise reduction requirements. The combination of structured pruning and mixed-precision quantization achieves "ultra-lightweight" model, further meeting the real-time noise reduction requirements of edge devices and balancing performance and deployment efficiency.

[0101] The audio noise reduction device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0102] The audio noise reduction device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0103] The audio noise reduction device provided in this application embodiment can implement the various processes implemented in the method embodiment, and will not be described again here to avoid repetition.

[0104] Optionally, such as Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601 and a memory 602. The memory 602 stores a program or instructions that can run on the processor 601. When the program or instructions are executed by the processor 601, they implement the various steps of the above-described audio noise reduction method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0105] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0106] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0107] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0108] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0109] The processor 710 can be used for: The first audio signal is subjected to an equivalent rectangular bandwidth transformation to obtain the first audio feature; The first audio feature is input into the audio denoising model, which includes an encoder, a processor, and a decoder. The encoder encodes the first audio feature to obtain an audio feature vector. The processor performs a convolution operation on the audio feature vector to obtain a convolutional feature vector. The decoder decodes the convolutional feature vector to obtain the second audio feature. The processor includes an enhanced temporal convolutional network block, which includes a causal dilated convolutional branch. The second audio feature is subjected to inverse equivalent rectangular bandwidth transformation to obtain the second audio signal.

[0110] In this way, the equivalent rectangular bandwidth transformation, based on the characteristics of the human auditory filter bank, preserves high resolution for low-frequency signals and reduces redundancy for high-frequency signals, making the extracted first audio features more consistent with the human auditory perception mechanism. After being input into the audio denoising model, the enhanced temporal convolutional network block captures long-term dependencies in the audio signal through causal dilation convolutional branches, while maintaining temporal causality, thus increasing the temporal receptive field and reducing inference latency. In this way, the denoising effect is improved under complex noise, audio distortion is reduced, and denoising efficiency is increased to meet the real-time requirements of voice interaction.

[0111] In some embodiments, the number of enhanced temporal convolutional network blocks is B, and each enhanced temporal convolutional network block includes K causal dilated convolutional branches; The output value of the b-th enhanced temporal convolutional network block is obtained by fusing the output values ​​of the K causal dilated convolutional branches of the b-th enhanced temporal convolutional network block with the output value of the (b-1)-th enhanced temporal convolutional network block; B and K are integers greater than 1, and b is a positive integer less than or equal to B.

[0112] In this way, multiple Enhanced-TCN blocks allow the model to learn more complex feature levels, improving noise reduction performance. Furthermore, Enhanced-TCN blocks effectively capture long temporal dependencies and multi-scale features through multi-branch causal dilated convolution and cross-block fusion, significantly increasing the temporal receptive field while significantly reducing inference latency.

[0113] In some embodiments, the processor 710 can also be used for: Acquire the raw audio signal; The original audio signal is subjected to inverse filtering and compensation processing to obtain the first audio signal.

[0114] In this way, the original audio signal may be distorted in frequency response due to the hardware filtering characteristics of the acquisition device. Inverse filtering compensation corrects the amplitude and phase frequency distortions introduced by the device, making the first audio signal input to subsequent processing closer to the characteristics of the real sound source, improving the accuracy of ERB transform features, and indirectly improving the input quality of the audio noise reduction model and the final audio noise reduction effect.

[0115] In some embodiments, the processor 710 can also be used for: Acquire sensing information collected by multimodal sensors; Determine scene information based on perceived information; Based on scene information, an audio noise reduction model corresponding to the scene information is determined; the model parameters of the audio noise reduction model are different for different scene information.

[0116] In this way, the noise characteristics of different scenarios vary significantly. By acquiring scene information through multimodal sensors and calling model parameters adapted to the scenario, the noise reduction strategy can be dynamically optimized, avoiding the effect fluctuation of a fixed model in complex scenarios and improving the robustness of noise reduction and scene adaptability.

[0117] In some embodiments, determining scene information based on perception information may include: The perceived information is input into the scene classifier, which then identifies the scene information.

[0118] In this way, using a scene classifier to automatically identify the environment type and determine the scene information can simplify the process and improve efficiency.

[0119] In some embodiments, the processor 710 can also be used for: Obtain multiple training samples; Construct an initial model; the first loss function of the initial model includes a perceptual loss function, a temporal envelope loss function, and a phase consistency loss function. The perceptual loss function is used to characterize the perceptual loudness error corresponding to the masking effect and frequency sensitivity. The temporal envelope loss function is used to ensure the macroscopic dynamics and energy fluctuations of the audio signal. The phase consistency loss function is used to penalize the phase spectrum inconsistency of the audio signal before and after noise reduction. The initial model is trained using multiple training samples until the initial model is determined to have converged based on the first loss function, thus obtaining the audio noise reduction model.

[0120] In this way, perceptual loss directly optimizes the perceived loudness error of denoised audio by simulating the masking effect and frequency sensitivity of the human ear. Temporal envelope loss reflects the macroscopic energy fluctuations of the audio; this constraint ensures that the dynamic range of the denoised audio is not overly smoothed, improving naturalness. Phase consistency loss reduces phase distortion by penalizing the difference in phase spectrum before and after denoising, ensuring phase consistency and improving the auditory realism of the audio. The three factors work synergistically to optimize the model training objective from the three dimensions of perception, dynamics, and phase, significantly improving the subjective experience of denoised audio.

[0121] In some embodiments, the processor 710 can also be used for: The initial model is trained based on multiple training samples until the initial model is determined to have converged based on the first loss function, at which point the first model is obtained. With the first model being the teacher model, a lightweight student model is constructed. A second loss function for the student model is generated based on the distillation loss function and the supervised loss function. The distillation loss function is used to represent the difference in feature distribution output by the teacher model and the student model in the intermediate layers of the network, and the supervised loss function is used to represent the difference in probability distribution output by the teacher model and the student model. The student model is trained by knowledge distillation based on multiple training samples and a second loss function to obtain an audio noise reduction model.

[0122] In this way, the first model, which usually has a large number of parameters and excellent performance but is difficult to deploy, can be used as the teacher model. The lightweight student model transfers the "hidden knowledge" of the teacher model through knowledge distillation. While maintaining performance close to that of the teacher model, the number of parameters and computation are greatly reduced. This solves the contradiction between high-performance models and low-computing-power devices and improves the practicality of audio noise reduction models.

[0123] In some embodiments, the processor 710 can also be used for: The student model is trained by knowledge distillation based on multiple training samples and the second loss function to obtain the second model. Determine the importance score for each convolutional channel in the second model; The convolutional channels in the second model with importance scores below the importance threshold are removed to obtain the audio noise reduction model.

[0124] In this way, redundant channels can be removed by evaluating the importance score of convolutional channels, thereby reducing model parameters and computational load while keeping performance loss under control and improving inference speed.

[0125] In some embodiments, the processor 710 can also be used for: The convolutional channels in the second model with importance scores lower than the importance threshold are removed to obtain the third model; The third model is subjected to mixed precision quantization to obtain the audio noise reduction model.

[0126] In this way, by quantizing model parameters from high precision to low precision, memory usage and computation time are reduced, while ensuring that the numerical accuracy meets the noise reduction requirements. The combination of structured pruning and mixed-precision quantization achieves "ultra-lightweight" model, further meeting the real-time noise reduction requirements of edge devices and balancing performance and deployment efficiency.

[0127] It should be understood that, in this embodiment, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 707 includes at least one of a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0128] The memory 709 can be used to store software programs and various data. The memory 709 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 709 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 709 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0129] Processor 710 may include one or more processing units; optionally, processor 710 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 710.

[0130] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described audio noise reduction method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0131] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0132] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described audio noise reduction method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0133] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0134] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the audio noise reduction method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0135] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0137] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An audio noise reduction method, characterized in that, The method includes: The first audio signal is subjected to an equivalent rectangular bandwidth transformation to obtain the first audio feature; The first audio feature is input into an audio denoising model, which includes an encoder, a processor, and a decoder. The encoder encodes the first audio feature to obtain an audio feature vector, the processor performs a convolution operation on the audio feature vector to obtain a convolutional feature vector, and the decoder decodes the convolutional feature vector to obtain a second audio feature. The processor includes an enhanced temporal convolutional network block, which includes a causal dilated convolutional branch. The second audio feature is subjected to inverse equivalent rectangular bandwidth transformation to obtain the second audio signal.

2. The method according to claim 1, characterized in that, The number of the enhanced temporal convolutional network blocks is B, and each enhanced temporal convolutional network block includes K causal dilated convolutional branches; The output value of the b-th enhanced temporal convolutional network block is obtained by fusing the output values ​​of the K causal dilated convolutional branches of the b-th enhanced temporal convolutional network block with the output value of the (b-1)-th enhanced temporal convolutional network block; B and K are integers greater than 1, and b is a positive integer less than or equal to B.

3. The method according to claim 1, characterized in that, Before performing the equivalent rectangular bandwidth transformation on the first audio signal to obtain the first audio features, the method further includes: Acquire the raw audio signal; The original audio signal is subjected to inverse filtering and compensation processing to obtain the first audio signal.

4. The method according to claim 1, characterized in that, Before inputting the first audio feature into the audio noise reduction model, the method further includes: Acquire sensing information collected by multimodal sensors; Determine scene information based on the perceived information; Based on the scene information, an audio noise reduction model corresponding to the scene information is determined; wherein, the model parameters of the audio noise reduction model corresponding to different scene information are different.

5. The method according to claim 4, characterized in that, Determining scene information based on the perceived information includes: The perceived information is input into a scene classifier, which then identifies the scene information.

6. The method according to any one of claims 1 to 5, characterized in that, Before inputting the first audio feature into the audio noise reduction model, the method further includes: Obtain multiple training samples; An initial model is constructed; the first loss function of the initial model includes a perceptual loss function, a temporal envelope loss function, and a phase consistency loss function. The perceptual loss function is used to characterize the perceptual loudness error corresponding to the masking effect and frequency sensitivity. The temporal envelope loss function is used to ensure the macroscopic dynamics and energy fluctuations of the audio signal. The phase consistency loss function is used to penalize the phase spectrum inconsistency of the audio signal before and after noise reduction. The initial model is trained based on the multiple training samples until the initial model is determined to have converged based on the first loss function, thus obtaining the audio noise reduction model.

7. The method according to claim 6, characterized in that, The step of training the initial model based on the multiple training samples until the initial model is determined to have converged based on the first loss function, to obtain the audio noise reduction model, includes: The initial model is trained based on the multiple training samples until the initial model is determined to have converged based on the first loss function, thus obtaining the first model. If the first model is a teacher model, a lightweight student model is constructed. A second loss function for the student model is generated based on the distillation loss function and the supervised loss function; the distillation loss function is used to represent the difference in feature distribution output by the teacher model and the student model in the intermediate layer of the network, and the supervised loss function is used to represent the difference in probability distribution output by the teacher model and the student model. The student model is trained by knowledge distillation based on the multiple training samples and the second loss function to obtain an audio noise reduction model.

8. The method according to claim 7, characterized in that, The step of training the student model using knowledge distillation based on the multiple training samples and the second loss function to obtain the audio noise reduction model includes: The student model is trained by knowledge distillation based on the multiple training samples and the second loss function to obtain the second model. Determine the importance score for each convolutional channel in the second model; The convolutional channels in the second model whose importance scores are less than the importance threshold are removed to obtain the audio noise reduction model.

9. The method according to claim 8, characterized in that, The convolutional channels in the second model with importance scores less than the importance threshold are removed to obtain the audio noise reduction model, which includes: The convolutional channels in the second model whose importance scores are less than the importance threshold are removed to obtain the third model; The third model is subjected to mixed precision quantization to obtain the audio noise reduction model.

10. An audio noise reduction device, characterized in that, The device includes: The first processing module is used to perform equivalent rectangular bandwidth transformation on the first audio signal to obtain the first audio features; The model processing module is used to input the first audio feature into an audio denoising model. The audio denoising model includes an encoder, a processor, and a decoder. The encoder encodes the first audio feature to obtain an audio feature vector. The processor performs a convolution operation on the audio feature vector to obtain a convolutional feature vector. The decoder decodes the convolutional feature vector to obtain a second audio feature. The processor includes an enhanced temporal convolutional network block, which includes a causal dilated convolutional branch. The second processing module is used to perform inverse equivalent rectangular bandwidth transformation on the second audio feature to obtain the second audio signal.

Images