Audio noise reduction method and device, electronic equipment and storage medium

Abstract

The invention provides an audio noise reduction method and device, electronic equipment and a storage medium, and belongs to the technical field of audio signal processing, and the method comprises the steps: extracting audio features in an original audio signal, and dividing the original audio signal into a plurality of audio clips based on the audio features; adding a position label and a noise label to each audio clip; performing real-time sampling on all the audio clips to obtain a plurality of digital audio signal clips, and performing pre-emphasis processing on each digital audio signal clip based on a noise tag to obtain a first audio; processing the first audio based on a bimodal feature extraction algorithm and a target noise reduction model to obtain a second audio, and obtaining a denoised target audio based on the second audio and the position tag; the dual modes comprise a time domain mode and a frequency domain mode, and the target noise reduction model is a trained noise reduction model. The definition and naturalness of the audio signal can be remarkably improved, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of audio signal processing, and more specifically, relates to an audio noise reduction method and device, an electronic device, and a storage medium. Background Art

[0002] Audio signal processing has extensive and important applications in many fields, including communications, multimedia, speech recognition, smart wearable devices, and virtual reality. However, in the actual audio acquisition and transmission process, audio signals are often inevitably interfered by various noises, which seriously affect the quality of audio signals and the performance of subsequent related applications. At present, although traditional audio noise reduction methods can suppress noise to a certain extent, they have many limitations. Some methods will mistakenly remove useful audio signal components while removing noise, resulting in distortion of audio signals and affecting the naturalness of the audio. Summary of the invention

[0003] The purpose of the present disclosure is to provide an audio noise reduction method and device, an electronic device, and a storage medium to significantly improve the clarity and naturalness of audio signals and enhance user experience.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an audio noise reduction method, comprising: Extracting audio features from an original audio signal, and dividing the original audio signal into a plurality of audio segments based on the audio features; Adding a position tag and a noise tag to each audio segment; performing real-time sampling on all audio segments to obtain a plurality of digital audio signal segments, and performing pre-emphasis processing on each digital audio signal segment based on the noise tag to obtain a first audio; The first audio is processed based on a bimodal feature extraction algorithm and a target noise reduction model to obtain a second audio, and the noise-reduced target audio is obtained based on the second audio and the position tag; the bimodality is a time domain and a frequency domain mode, and the target noise reduction model is a trained noise reduction model.

[0005] According to a second aspect of the embodiments of the present disclosure, there is provided an audio noise reduction device, comprising: A feature extraction module, used to extract audio features from an original audio signal, and divide the original audio signal into a plurality of audio segments based on the audio features; A preprocessing module is used to add a position tag and a noise tag to each audio segment; perform real-time sampling on all audio segments to obtain multiple digital audio signal segments, and perform pre-emphasis processing on each digital audio signal segment based on the noise tag to obtain a first audio; The denoising module is used to process the first audio based on a bimodal feature extraction algorithm and a target denoising model to obtain a second audio, and to obtain a denoised target audio based on the second audio and the position tag; the bimodality is time domain and frequency domain, and the target denoising model is a trained denoising model.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned audio noise reduction method when executing the computer program.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned audio noise reduction method are implemented.

[0008] The audio noise reduction method and device, electronic device, and storage medium provided by the embodiments of the present disclosure have the following beneficial effects: On the one hand, the present disclosure can significantly improve the audio noise reduction effect. Audio features are extracted from the original audio signal and audio segments are divided. The digital audio signal segments are pre-emphasized according to the noise labels, which effectively enhances the high-frequency signal and improves the high-frequency signal-to-noise ratio of the signal, laying the foundation for subsequent noise reduction. On top of this, a dual-modal feature extraction algorithm is used to comprehensively capture audio features by combining information in the time domain and frequency domain, and then processed using a trained target noise reduction model. This can accurately remove all types of noise and restore clear and pure target audio, greatly improving the quality and audibility of the audio, and can bring users a high-quality audio experience whether in voice communication, multimedia playback or audio recording.

[0009] On the other hand, the process of processing audio disclosed in the present invention is efficient and accurate. After dividing the audio segments and adding position and noise labels, the audio signal can be sampled in real time to obtain digital signal segments. Pre-emphasis processing, dual-modal feature extraction, and target noise reduction model cooperate with each other to optimize and reduce the noise of the audio step by step in an orderly manner. This processing method not only reduces unnecessary waste of computing resources and improves processing efficiency, but also ensures the accuracy of noise reduction through precise feature extraction and model processing, so that the final target audio is far superior in quality to traditional simple noise reduction methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A flowchart of an audio noise reduction method provided by an embodiment of the present disclosure; Figure 2 A structural block diagram of an audio noise reduction device provided by an embodiment of the present disclosure; Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0013] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0014] Please refer to Figure 1 , Figure 1 A flowchart of an audio noise reduction method provided by an embodiment of the present disclosure is provided, and the method includes: S101: extracting audio features from an original audio signal, and dividing the original audio signal into a plurality of audio segments based on the audio features.

[0015] In this embodiment, the original audio signal refers to an audio signal directly obtained from an audio source (such as a microphone) without any specific processing (such as noise reduction, enhancement, etc.). It contains various audio components, including the speaker's voice, the melody of music, and background noise. For example, a piece of audio waveform data containing human voice and vehicle driving sound recorded by a microphone on a noisy street is the original audio signal.

[0016] Audio features are information extracted from audio signals that can characterize certain characteristics of audio signals. These features can be used to distinguish different types of audio signals, such as basic features such as pitch, timbre, and loudness. Audio features can also include time domain features and frequency domain features. Time domain features such as zero crossing rate, which reflects the number of times a signal crosses zero in a unit of time, can be used to distinguish between speech and noise, because the zero crossing rate of speech signals is relatively regular, while the zero crossing rate of noise is relatively random. Frequency domain features, such as Mel-Frequency Cepstral Coefficients, or MFCC for short, can mimic the human ear's perception of sounds of different frequencies and are commonly used features in fields such as speech recognition.

[0017] An audio clip is a portion of the original audio signal divided according to certain rules. Each audio clip has a certain consistency or correlation in audio content or characteristics. For example, based on the changes in audio features, an original audio clip containing dialogue and music can be divided into "dialogue audio clip" and "music audio clip". Such division helps to carry out targeted processing of different types of audio content, such as voice noise reduction processing for dialogue audio clips and timbre adjustment for music audio clips.

[0018] In this embodiment, extracting audio features from the original audio signal includes: extracting time domain features and frequency domain features from the original audio signal using short-time Fourier transform; or extracting multi-resolution time-frequency features from the original audio signal using discrete wavelet transform; or extracting prediction coefficients of the audio signal using linear prediction coding. The time domain features may be short-time energy, short-time average amplitude, zero-crossing rate, etc.

[0019] Divide the original audio signal into multiple audio segments based on audio features, including: divide the original audio signal into multiple audio segments based on a zero-crossing rate threshold. Divide the original audio signal into frames of fixed length, each frame is N sampling points, the frame shift is M sampling points, and N and M are both positive integers M≤N. Calculate the zero-crossing rate of each frame. Set the zero-crossing rate threshold to T. When the zero-crossing rate of a frame is greater than T, it is determined that the frame and subsequent frames greater than T belong to the same audio segment; when the zero-crossing rate of a frame is less than or equal to T, it is determined that the frame and subsequent frames less than or equal to T belong to the same audio segment.

[0020] S102: adding a position tag and a noise tag to each audio segment; performing real-time sampling on all audio segments to obtain a plurality of digital audio signal segments, and performing pre-emphasis processing on each digital audio signal segment based on the noise tag to obtain a first audio.

[0021] In this embodiment, the position tag is identification information added to each audio segment, which is used to indicate the position of the audio segment in the original audio signal. This position can be a time position, such as from the xth second to the yth second from the start of the original audio; or it can be a position based on audio sampling points, such as from the mth sampling point to the nth sampling point. The position tag helps to accurately locate and operate a specific audio segment in subsequent processing, and facilitates the merging of the denoised audio content.

[0022] The noise label is a descriptive identifier of the noise situation in each audio clip. It can indicate whether the audio clip contains noise, the type of noise (such as white noise, pink noise, traffic noise, etc.), the intensity level of the noise, and other information.

[0023] Real-time sampling refers to the continuous sampling of audio signals at certain time intervals during audio processing. When real-time sampling is performed on all audio clips, a series of discrete sample values ​​are obtained on each audio clip according to the set sampling frequency, and the continuous audio signal is converted into a discrete digital signal, thereby obtaining multiple digital audio signal clips.

[0024] A digital audio signal segment is a discrete form of audio data obtained from an audio segment through real-time sampling. Each digital audio signal segment consists of a series of sample values ​​arranged in sampling order. These sample values ​​are usually stored in the form of binary numbers, representing the amplitude information of the audio signal at different times.

[0025] In this embodiment, a position label and a noise label are added to each audio segment; all audio segments are sampled in real time to obtain multiple digital audio signal segments. It represents an identification information assigned to each audio segment, which is used to clarify the specific position of the segment in the original audio signal. This position can be represented from different dimensions, the most common of which is the time dimension. For example, if an audio segment lasts from the 3rd second to the 8th second after the start of the original audio, then "3 seconds-8 seconds" is its time position label; from the perspective of audio sampling points, if the audio sampling frequency is fixed, the sampling point sequence number can also be used to represent the position, for example, from the 1000th sampling point to the 3000th sampling point, "1000-3000" is the corresponding position label. Adding a noise label to each audio segment can indicate whether there is noise in the audio segment and the specific characteristics of the noise. Real-time sampling of each audio segment can obtain multiple digital audio signal segments.

[0026] Pre-emphasis is an audio signal processing technology whose main purpose is to enhance the strength of high-frequency signals. In audio signals, the high-frequency part is usually relatively weak and easily affected by noise. Pre-emphasis increases the amplitude of high-frequency components by filtering digital audio signal segments, thereby improving the high-frequency signal-to-noise ratio of the signal and improving audio quality.

[0027] Pre-emphasis processing is performed on each digital audio signal segment based on the noise label to obtain the first audio. This means that according to the noise label information carried by each segment, a suitable pre-emphasis parameter (such as a pre-emphasis coefficient) is selected in a targeted manner, and a pre-emphasis operation is performed on the corresponding digital audio signal segment. The audio obtained after such processing is the first audio.

[0028] S103: Process the first audio based on a bimodal feature extraction algorithm and a target noise reduction model to obtain a second audio, and obtain a noise-reduced target audio based on the second audio and the position tag; the bimodalities are time domain and frequency domain modes, and the target noise reduction model is a trained noise reduction model.

[0029] In this embodiment, dual-modality refers to two modes: time domain and frequency domain. Dual-modal feature extraction algorithm: is an algorithm that can extract features from the time domain and frequency domain of the audio signal at the same time. In the time domain, features such as short-time energy and zero-crossing rate are extracted. These features can help distinguish between speech and noise, judge the activity of speech, etc. In the frequency domain, features such as spectrum peaks, frequency band energy distribution, and Mel-frequency cepstral coefficients (MFCC) are extracted. These features are very useful for analyzing the timbre, pitch, and intensity of different frequency components of the audio signal.

[0030] The target denoising model is a trained denoising model that is trained with a large amount of noisy audio data (training data). During the training process, the model learns the characteristic patterns of different types of noise and how to restore pure audio from noisy audio. The training data usually includes known pure audio and corresponding audio with different types and intensities of noise. The model adjusts its own parameters by comparing the differences between them, so as to achieve the purpose of effectively denoising new noisy audio in practical applications.

[0031] The second audio is the audio obtained after the first audio is processed by the bimodal feature extraction algorithm and the features are then input into the target noise reduction model for processing. Compared with the first audio, the second audio has less noise components and improved audio quality. The second audio includes multiple audio clips after noise reduction processing. It is the intermediate result audio after the preliminary noise reduction processing and needs to be further processed in combination with the position tag to generate the final target audio.

[0032] In this embodiment, when performing audio processing, the first object to be processed is the first audio, which is the audio after pre-emphasis processing, but there is still a lot of noise. For example, the first audio can be feature extracted from both the time domain and the frequency domain to fully capture the characteristics of the audio signal in terms of time change and frequency distribution. Then, the extracted bimodal features are input into the target noise reduction model that has been pre-trained with a large amount of data. The model analyzes and processes the input features based on the learned feature differences between the noise and the pure audio, thereby outputting the noise-reduced audio, i.e., the second audio. Although the second audio has been processed for noise reduction at this time, it is still in a fragmented state. It is necessary to splice and integrate these fragments according to their correct order and position in the original audio based on the position tags previously added to each audio fragment, so as to finally obtain a complete target audio without excess noise and maintaining content coherence.

[0033] It can be concluded from the above that, on the one hand, the present disclosure can significantly improve the audio noise reduction effect. Audio features are extracted from the original audio signal and the audio segments are divided. The digital audio signal segments are pre-emphasized according to the noise labels, which effectively enhances the high-frequency signal and improves the high-frequency signal-to-noise ratio of the signal, laying the foundation for subsequent noise reduction. On top of this, a dual-modal feature extraction algorithm is used to comprehensively capture audio features by combining information in the time domain and frequency domain, and then processed using a trained target noise reduction model. This can accurately remove all types of noise and restore clear and pure target audio, greatly improving the quality and audibility of the audio, and can bring users a high-quality audio experience whether in voice communication, multimedia playback or audio recording.

[0034] On the other hand, the process of processing audio disclosed in the present invention is efficient and accurate. After dividing the audio segments and adding position and noise labels, the audio signal can be sampled in real time to obtain digital signal segments. Pre-emphasis processing, dual-modal feature extraction, and target noise reduction model cooperate with each other to optimize and reduce the noise of the audio step by step in an orderly manner. This processing method not only reduces unnecessary waste of computing resources and improves processing efficiency, but also ensures the accuracy of noise reduction through precise feature extraction and model processing, so that the final target audio is far superior in quality to traditional simple noise reduction methods.

[0035] In one embodiment of the present disclosure, the first audio is processed based on a bimodal feature extraction algorithm and a target noise reduction model to obtain a second audio, including: Extracting features from the first audio using a dual-modal feature extraction algorithm based on time domain and frequency domain to obtain dual-modal features; The first audio and the bimodal features are input into a target noise reduction model for processing to obtain a second audio.

[0036] In this embodiment, the bimodal feature extraction algorithm combines information of two or more different modes to extract audio features from the original data. For example, the bimodal feature extraction algorithm includes three steps: time domain feature extraction, frequency domain feature extraction, and feature fusion. By combining the features extracted from the time domain (such as short-time energy, short-time average amplitude, zero crossing rate, etc.) and the features extracted from the frequency domain (such as spectrum amplitude, MFCC, etc.), bimodal features based on the time domain and frequency domain are obtained. These features can be represented in the form of vectors. For example, the time domain features and frequency domain features of a frame of audio are arranged in sequence to form a feature vector. The feature vectors of all frames constitute a bimodal feature sequence, and the bimodal feature sequence can be input into the target noise reduction model for noise reduction.

[0037] The training phase of the target noise reduction model requires the collection of a large amount of noisy audio data and their corresponding clean audio data. These data cover various noise types (such as white noise, traffic noise, human voice noise, etc.) and different audio scenes (such as indoor, outdoor, phone calls, etc.). The noisy audio data is used as the input model, while the clean audio data is used as the target output of model learning. The same bimodal feature extraction algorithm as that used to process the first audio is applied to the collected noisy audio data to obtain the corresponding bimodal features. These features will be used as the input for model training together with the noisy audio data.

[0038] The target noise reduction model is trained using the collected noisy audio data and its bimodal features as input and the corresponding clean audio data as output. During the training process, the model continuously adjusts its own parameters to learn the mapping relationship between the noisy audio data and bimodal features and the clean audio, so that the model can predict the clean audio as accurately as possible based on the input noisy audio and bimodal features. Common training methods include back propagation algorithms (used in neural network models), etc., which optimize model parameters by minimizing the difference between the predicted output and the actual clean audio (such as loss functions such as mean square error). Finally, after the model training is completed, the first audio and bimodal features are input into the target noise reduction model for processing to obtain the second audio.

[0039] It can be concluded from the above that this embodiment uses a dual-modal feature extraction algorithm and a target noise reduction model to process the first audio, which significantly improves the audio noise reduction effect. By fusing time domain and frequency domain features, the key information of the audio signal is captured more comprehensively, so that the noise reduction model can more accurately identify and remove noise. This method not only improves the accuracy and efficiency of noise reduction, but also ensures the high fidelity of the second audio.

[0040] In one embodiment of the present disclosure, performing pre-emphasis processing on each digital audio signal segment based on a noise label to obtain a first audio includes: An initial weight coefficient of a filter corresponding to each digital audio signal segment and an adjustment parameter corresponding to the weight are determined based on the noise label.

[0041] In this embodiment, during the pre-emphasis process, a filter is required to process the digital audio signal segment. The weight coefficient of the filter determines the degree of influence of the filter on different frequency components. The initial weight coefficient refers to the weight parameter of the filter when the digital audio signal segment is filtered in the initial state. These coefficients can be determined according to the noise label. For example, for audio segments with high noise intensity, different initial weight coefficients are set to more effectively enhance high-frequency signals.

[0042] In addition to the initial weight coefficients, there are also adjustment parameters corresponding to the weights. These adjustment parameters are used to dynamically adjust the weight coefficients of the filter during the pre-emphasis process according to the actual situation of the audio signal and the processing requirements. For example, depending on the type and intensity of the noise, the adjustment parameters can control the change amplitude and direction of the weight coefficient, thereby finely adjusting the degree of enhancement of the high-frequency signal.

[0043] It can be concluded from the above that the embodiment automatically adjusts the filter according to the noise intensity, which can effectively enhance the high-frequency signal while maintaining the naturalness of the audio. Dynamic adjustment of parameters ensures the flexibility of the processing process to adapt to audio signals of different noise types and intensities, thereby improving the pertinence and efficiency of audio processing.

[0044] In one embodiment of the present disclosure, the adjustment parameters corresponding to the weights include a step size factor and a filter order; Determining an initial weight coefficient of a filter corresponding to each digital audio signal segment and an adjustment parameter corresponding to the weight based on the noise label includes: In response to the noise type corresponding to the noise label belonging to the first noise type, determining that the value range of the step factor is a first value range, and the order of the filter is less than or equal to 5; In response to the noise type corresponding to the noise label belonging to the second noise type, determining that the value range of the step factor is a second value range, and the order of the filter is greater than 5; The frequency components of the second noise type are greater than the frequency components of the first noise type.

[0045] In this embodiment, the adjustment parameters corresponding to the weights include the step factor and the filter order. The step factor determines the magnitude of the filter weight update in each iteration. A larger step factor can quickly update the weights, speed up the convergence of the algorithm, and make the filter output close to the expected signal in a short time; a smaller step factor will reduce the weight update magnitude and slow down the convergence, but make the algorithm more stable, and the final steady-state error will be smaller.

[0046] In response to the noise type corresponding to the noise label belonging to the first type of noise, the value range of the step factor is determined to be (0, 0.6). In response to the noise type corresponding to the noise label belonging to the second type of noise, the value range of the step factor is determined to be (0.6, 1). The value range of the step factor is generally between (0, 1). If the value is too large, the algorithm converges quickly, but oscillation or even divergence may occur, resulting in failure to converge to the optimal solution; if the value is too small, the algorithm stability is improved, but the convergence process will become long, which cannot meet the requirements in scenarios with high real-time requirements.

[0047] The filter order refers to the number of weight coefficients in the filter, which indicates the complexity of the filter and its ability to process signals. A higher filter order means that the filter has stronger fitting ability and can process more complex signals and noise characteristics; while a lower filter order makes the filter structure simpler and requires less calculation. If the filter order is too low, the filter may not be able to fully capture the signal characteristics, and it is difficult to achieve the ideal filtering effect, effectively suppress noise or approach the expected signal; on the contrary, if the order is too high, although it can theoretically improve performance, it will significantly increase the amount of calculation and storage requirements, and will also lead to overfitting problems, that is, the filter fits the training data well, but has poor generalization ability for new data.

[0048] In this embodiment, when the noise type belongs to the first type of noise, the noise spectrum of its signal is relatively simple and has fewer frequency components, and the order of the filter can be set to a positive integer less than or equal to 5. When the noise type belongs to the second type of noise, the frequency components of its signal are relatively complex, and the order of the filter can be set to a positive integer greater than 5. At this time, the corresponding mid-order filter can provide better frequency selectivity and transition band characteristics, and can better suppress noise and interference without bringing excessive calculation burden.

[0049] It can be concluded from the above that this embodiment achieves effective processing of noise with different frequency components by adjusting the step size factor and the filter order according to the noise type, which not only ensures the rapid convergence and stability of the algorithm, but also avoids the overfitting problem, improves the performance of the filter, and achieves good pre-emphasis processing.

[0050] In one embodiment of the present disclosure, the training process of the target denoising model includes: Determine the multi-task learning strategy, main loss function and auxiliary loss function; The denoising model is trained based on the multi-task learning strategy, the main loss function, the auxiliary loss function and the audio signal in the audio library to obtain the target denoising model.

[0051] In this embodiment, multi-task learning is a machine learning method that allows the model to learn multiple related tasks at the same time to improve the generalization and performance of the model. In the training of the target noise reduction model, the multi-task learning strategy involves letting the model handle multiple noise reduction-related tasks at the same time, such as not only removing noise from the audio, but also optimizing other features of the audio (such as the clarity of speech, the timbre of music, etc.). By learning these tasks simultaneously, the model can extract complementary information from different tasks, thereby better understanding the audio data and improving the noise reduction effect.

[0052] The loss function is used to measure the difference between the model's prediction and the actual result. The main loss function is the loss function that plays a major role in the model training process. It is directly related to the main task of the model. For example, in the noise reduction task, the main loss function can be the mean squared error (MSE) between the predicted noise-reduced audio and the original clean audio. By minimizing the main loss function, the model can continuously adjust the parameters to make the predicted noise-reduced audio as close to the original clean audio as possible.

[0053] The auxiliary loss function is introduced to help the main loss function better train the model. It is usually related to some auxiliary tasks of the model, which can provide additional information to help the model better learn the characteristics of audio data. For example, the auxiliary loss function can be used to constrain the model to learn certain specific features of audio, such as the spectral characteristics and time domain characteristics of audio, so as to enhance the generalization ability and stability of the model.

[0054] The audio library is a dataset containing a large number of audio signals, which can include various types of noisy audio and corresponding clean audio. During the training process, these audio signals are used as input data for the model. The model continuously adjusts its parameters by learning from these data to adapt to different audio scenarios and noise types, thereby improving the model's noise reduction performance.

[0055] In this embodiment, the multi-task learning strategy, the main loss function and the auxiliary loss function are first determined. After the above elements are determined, the denoising model is trained using the audio signals in the audio library. During the training process, the model optimizes the main loss function and the auxiliary loss function at the same time according to the multi-task learning strategy. By continuously adjusting the parameters of the model, the model can meet the requirements of the main loss function when predicting the denoised audio, and be as close to the original pure audio as possible, and can also learn other useful features of the audio through the auxiliary loss function, thereby improving the generalization ability of the model. After multiple iterations of training, a target denoising model with good performance is finally obtained.

[0056] From the above, it can be concluded that this embodiment fully utilizes the audio library resources to train the noise reduction model through the combination of multi-task learning strategy and main and auxiliary loss functions. This method not only improves the noise reduction effect, but also enhances the generalization ability of the model, so that it can better adapt to a variety of audio scenes and noise types, and effectively improves the performance and stability of the target noise reduction model.

[0057] In one embodiment of the present disclosure, dividing an original audio signal into a plurality of audio segments based on audio features includes: An initial length of the audio segment is adjusted based on the entropy value of the audio feature to obtain a first length, and an original audio signal is divided into a plurality of audio segments based on the first length.

[0058] Entropy is an indicator used to measure the uncertainty or confusion of a system. In this embodiment, the entropy value of the audio feature indicates the uncertainty of the information contained in the audio feature. For example, if the entropy value of the audio feature is high, it means that the changes of the audio signal in this feature dimension are more complex and disordered, and the amount of information contained is large; conversely, if the entropy value is low, it means that the audio signal is relatively stable and regular in this feature, and the amount of information is small. For example, for a segment of audio containing a variety of different voices and background noise, the entropy value of its audio feature (such as spectral feature) is high because the frequency components of the signal are complex and changeable; while for a segment of audio with a single pure tone, the entropy value of its spectral feature is low because the frequency components are relatively single. The first length is the length obtained after adjusting the initial length of the audio segment. The basis for the adjustment is the entropy value of the audio feature, that is, the length of the audio segment is changed according to the uncertainty of the audio signal in a specific audio feature dimension. The higher the entropy value of the audio feature, the more complex the changes in the content of the audio part are, and a more refined division is required. At this time, the first length needs to be shorter than the initial length; conversely, if the entropy value is low and the content is relatively stable, the first length can be appropriately extended.

[0059] It can be concluded from the above that this embodiment realizes intelligent division of audio segments by dynamically adjusting the initial length of the audio segment based on the entropy value of the audio feature.

[0060] In an embodiment of the present disclosure, adjusting the initial length of the audio segment to obtain a first length based on the entropy value of the audio feature includes: When the entropy value of the audio feature is higher than a first preset threshold, shortening the initial length of the audio segment to obtain a first length; When the entropy value of the audio feature is lower than a second preset threshold, extending the initial length of the audio segment to obtain a first length; Among them, the first preset threshold is greater than the second preset threshold.

[0061] In this embodiment, the initial length refers to the length of the preset audio segment before any adjustment is made, and this length can be set based on specific standards or experience. The first and second preset thresholds are judgment criteria. When the entropy value of the audio feature is higher than the first preset threshold, it indicates that the content changes of the audio signal in this part are very complex, and contain many different types of sounds (such as complex voice conversations, multiple instruments playing at the same time, strong changes in environmental noise, etc.). In order to facilitate subsequent processing of the divided audio segments, the initial length needs to be shortened. For example, the original division according to the initial length of 1 second can now be shortened to a segment of 0.5 seconds, so that the audio content in each segment will not be too complex and diverse.

[0062] When the entropy value of the audio feature is lower than the second preset threshold, it means that the audio signal is relatively stable in this part and the content is relatively simple. At this time, extending the initial length can reduce the number of segments without affecting the requirements of subsequent processing for audio consistency. For example, extending the initial length from 0.5 seconds to 1 second can still ensure that the content of each audio segment is similar, and can also improve processing efficiency to a certain extent.

[0063] It can be concluded from the above that this embodiment effectively improves the flexibility and efficiency of audio processing by dynamically adjusting the length of the audio segment based on the audio feature entropy value. When the audio complexity is high, the segment is shortened to facilitate detailed analysis; when the audio is stable, the segment is extended to reduce the number of processing times. This adaptive adjustment method optimizes the audio processing flow and improves the overall performance.

[0064] Corresponding to the audio noise reduction method of the above embodiment, Figure 2 This is a structural block diagram of an audio noise reduction device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 2 The audio noise reduction device 20 includes: a feature extraction module 21, a preprocessing module 22 and a noise reduction module 23.

[0065] The feature extraction module 21 is used to extract audio features from the original audio signal and divide the original audio signal into multiple audio segments based on the audio features; The preprocessing module 22 is used to add a position tag and a noise tag to each audio segment; perform real-time sampling on all audio segments to obtain multiple digital audio signal segments, and perform pre-emphasis processing on each digital audio signal segment based on the noise tag to obtain a first audio; The denoising module 23 is used to process the first audio based on a bimodal feature extraction algorithm and a target denoising model to obtain a second audio, and to obtain a denoised target audio based on the second audio and a position tag; the bimodalities are time domain and frequency domain, and the target denoising model is a trained denoising model.

[0066] In one embodiment of the present disclosure, the noise reduction module 23 is specifically used to: Extracting features from the first audio using a dual-modal feature extraction algorithm based on time domain and frequency domain to obtain dual-modal features; The first audio and the bimodal features are input into a target noise reduction model for processing to obtain a second audio.

[0067] In one embodiment of the present disclosure, the preprocessing module 22 is specifically used for: An initial weight coefficient of a filter corresponding to each digital audio signal segment and an adjustment parameter corresponding to the weight are determined based on the noise label.

[0068] In one embodiment of the present disclosure, the adjustment parameters corresponding to the weights include a step size factor and a filter order; the preprocessing module 22 is specifically used for: In response to the noise type corresponding to the noise label belonging to the first noise type, determining that the value range of the step factor is a first value range, and the order of the filter is less than or equal to 5; In response to the noise type corresponding to the noise label belonging to the second noise type, determining that the value range of the step factor is a second value range, and the order of the filter is greater than 5; The frequency components of the second noise type are greater than the frequency components of the first noise type.

[0069] In one embodiment of the present disclosure, the noise reduction module 23 is specifically used to: Determine the multi-task learning strategy, main loss function and auxiliary loss function; The denoising model is trained based on the multi-task learning strategy, the main loss function, the auxiliary loss function and the audio signal in the audio library to obtain the target denoising model.

[0070] In one embodiment of the present disclosure, the feature extraction module 21 is specifically used for: An initial length of the audio segment is adjusted based on the entropy value of the audio feature to obtain a first length, and an original audio signal is divided into a plurality of audio segments based on the first length.

[0071] In one embodiment of the present disclosure, the feature extraction module 21 is specifically used for: When the entropy value of the audio feature is higher than a first preset threshold, shortening the initial length of the audio segment to obtain a first length; When the entropy value of the audio feature is lower than a second preset threshold, extending the initial length of the audio segment to obtain a first length; Among them, the first preset threshold is greater than the second preset threshold.

[0072] See also Figure 3 , Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 3 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of each module / unit in the above-mentioned device embodiments, such as Figure 2 The functions of modules 21 to 23 are shown.

[0073] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0074] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0075] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0076] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the audio noise reduction method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0077] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0078] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0079] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0081] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0083] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0084] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An audio noise reduction method, characterized in that: include: Extracting audio features from an original audio signal, and dividing the original audio signal into a plurality of audio segments based on the audio features; Adding a position tag and a noise tag to each audio segment; performing real-time sampling on all audio segments to obtain a plurality of digital audio signal segments, and performing pre-emphasis processing on each digital audio signal segment based on the noise tag to obtain a first audio; Processing the first audio based on a bimodal feature extraction algorithm and a target noise reduction model to obtain a second audio, and obtaining a noise-reduced target audio based on the second audio and the position tag; The dual modalities are time domain and frequency domain, and the target noise reduction model is a trained noise reduction model.

2. The audio noise reduction method according to claim 1, characterized in that: The step of processing the first audio based on the bimodal feature extraction algorithm and the target noise reduction model to obtain the second audio includes: Using a dual-modal feature extraction algorithm based on time domain and frequency domain to extract features from the first audio to obtain dual-modal features; The first audio and the bimodal features are input into a target noise reduction model for processing to obtain a second audio.

3. The audio noise reduction method according to claim 1, wherein: The pre-emphasis processing is performed on each digital audio signal segment based on the noise label to obtain the first audio, comprising: An initial weight coefficient of a filter corresponding to each digital audio signal segment and an adjustment parameter corresponding to the weight are determined based on the noise label.

4. The audio noise reduction method according to claim 3, characterized in that: The adjustment parameters corresponding to the weights include a step size factor and a filter order; Determining an initial weight coefficient of a filter corresponding to each digital audio signal segment and an adjustment parameter corresponding to the weight based on the noise label includes: In response to the noise type corresponding to the noise label belonging to the first noise type, determining that the value range of the step factor is a first value range, and the order of the filter is less than or equal to 5; In response to the noise type corresponding to the noise label belonging to the second noise type, determining that the value range of the step factor is a second value range, and the order of the filter is greater than 5; The frequency components of the second noise type are greater than the frequency components of the first noise type.

5. The audio noise reduction method according to claim 1, wherein: The training process of the target denoising model includes: Determine the multi-task learning strategy, main loss function and auxiliary loss function; The denoising model is trained based on the multi-task learning strategy, the main loss function, the auxiliary loss function and the audio signal in the audio library to obtain a target denoising model.

6. The audio noise reduction method according to claim 1, wherein: The step of dividing the original audio signal into a plurality of audio segments based on the audio feature comprises: An initial length of the audio segment is adjusted based on the entropy value of the audio feature to obtain a first length, and the original audio signal is divided into a plurality of audio segments based on the first length.

7. The audio noise reduction method according to claim 6, characterized in that: The adjusting the initial length of the audio segment based on the entropy value of the audio feature to obtain a first length includes: When the entropy value of the audio feature is higher than a first preset threshold, shortening the initial length of the audio segment to obtain a first length; When the entropy value of the audio feature is lower than a second preset threshold, extending the initial length of the audio segment to obtain a first length; Wherein, the first preset threshold is greater than the second preset threshold.

8. An audio noise reduction device, characterized in that: include: A feature extraction module, used to extract audio features from an original audio signal, and divide the original audio signal into a plurality of audio segments based on the audio features; A preprocessing module is used to add a position tag and a noise tag to each audio segment; perform real-time sampling on all audio segments to obtain multiple digital audio signal segments, and perform pre-emphasis processing on each digital audio signal segment based on the noise tag to obtain a first audio; The denoising module is used to process the first audio based on a bimodal feature extraction algorithm and a target denoising model to obtain a second audio, and to obtain a denoised target audio based on the second audio and the position tag; the bimodality is time domain and frequency domain, and the target denoising model is a trained denoising model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images