Howling suppression method, electronic equipment and computer readable storage medium

Abstract

The invention relates to the technical field of audio processing, and provides a howling suppression method, electronic equipment and a computer readable storage medium. The howling suppression method comprises the following steps: acquiring audio data; performing howling suppression pretreatment on the audio data, and inputting the audio data into a howling suppression network to obtain a howling suppression mask; performing howling detection pre-processing on the audio data, and inputting the processed audio data into a howling detection network to obtain howling detection output; weighting the howling suppression mask according to the howling detection output to obtain a weighted howling suppression mask; and performing howling suppression post-processing on the audio data according to the weighted howling suppression mask to obtain a howling suppression output audio. According to the method, the optimized howling suppression mask is used for performing howling suppression on the audio data, so that the problem of audio distortion output by howling suppression is avoided.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio processing technology, and in particular to a howling suppression method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In a sound reinforcement system composed of microphones, power amplifiers and speakers, the microphones and speakers form a closed loop of speaker-power amplifier-speaker, and certain frequencies are continuously strengthened, thus causing howling. When howling occurs, it will damage the speakers, impair hearing, and affect the user experience.

[0003] The prior art performs howling suppression by using a howling suppression algorithm. Among them, the howling suppression algorithm based on deep learning learns the characteristics of the signal and the statistical laws of howling through a deep neural network model, and performs howling suppression based on the howling detection results. Howling is an occasional event, and the howling suppression effect is closely related to the accuracy of howling detection. The prior art uses a method of howling detection through amplitude and frequency thresholds, which is easy to misdetect high-amplitude and high-frequency non-howling signals, resulting in audio distortion after howling suppression. Summary of the invention

[0004] In view of the above problems, the present application provides a howling suppression method, an electronic device and a computer-readable storage medium, which are used to solve the problem of howling suppression output audio distortion caused by misdetection of a howling detection algorithm.

[0005] According to one aspect of the present application, a howling suppression method is provided, the method comprising: acquiring audio data; performing howling suppression pre-processing on the audio data and inputting the audio data into a howling suppression network to obtain a howling suppression mask; performing howling detection pre-processing on the audio data and inputting the audio data into a howling detection network to obtain a howling detection output; weighting the howling suppression mask according to the howling detection output to obtain a weighted howling suppression mask; performing howling suppression post-processing on the audio data according to the weighted howling suppression mask to obtain a howling suppression output audio.

[0006] In an optional manner, weighting the howling suppression mask according to the howling detection output to obtain a weighted howling suppression mask includes: calculating a howling weight according to the howling detection output, a howling detection sensitivity control constant, and a howling suppression sensitivity control constant; weighting the howling suppression mask according to the howling weight to obtain a weighted howling suppression mask.

[0007] In an optional manner, the value range of the howling detection sensitivity constant and the howling suppression sensitivity constant are both 0 to 1; the calculating the howling weight according to the howling detection output, the howling detection sensitivity constant and the howling suppression sensitivity constant includes: if the sum of the howling detection output and the howling detection sensitivity constant is less than the howling suppression sensitivity constant, determining the sum of the howling detection output and the howling detection sensitivity constant as the howling weight; if the sum of the howling detection output and the howling detection sensitivity constant is greater than or equal to the howling suppression sensitivity constant, determining the howling suppression sensitivity constant as the howling weight.

[0008] In an optional manner, weighting the howling suppression mask according to the howling weight to obtain a weighted howling suppression mask includes: calculating the weighted howling suppression mask according to the following formula: hs_mask_weighted=howl_seight*hs_mask+1-howl_weight; wherein hs_mask_weighted is the weighted howling suppression mask, howl_weight is the howling weight, and hs_mask is the howling suppression mask.

[0009] In an optional manner, the howling detection network includes a feature extraction module, a recurrent neural network layer and a detection output module; the audio data is input into the howling detection network after being pre-processed for howling detection to obtain a howling detection output, including: converting the audio data from the time domain to the frequency domain frame by frame, obtaining the amplitude of each frame, calculating the logarithmic Mel spectrum of each frame according to the amplitude of each frame, and splicing the logarithmic Mel spectrum of each frame to obtain a splicing feature, wherein the dimensions of the splicing feature include data batch, feature channel, frame length and number of frames, and the feature channel includes an amplitude channel; the feature extraction module uses two-dimensional convolution to perform a sliding window operation on the splicing feature in two dimensions of frame length and frame number at the same time to obtain a third feature; the recurrent neural network layer extracts the third feature according to the temporal context information of the third feature to obtain a fourth feature; the detection output module uses a fully connected layer to reduce the dimension of the fourth feature in the channel dimension, and obtains a howling prediction probability value through an activation layer.

[0010] In an optional manner, the howling suppression pre-processing is performed by a howling suppression pre-processing module, and the howling suppression post-processing is performed by a howling suppression post-processing module. The howling suppression pre-processing module, the howling suppression network and the howling suppression post-processing module constitute a howling suppression model, and the processing process of the sample data of the howling suppression model includes: obtaining clean audio and impulse response; segmenting the clean audio to obtain segment audio signals; inputting the segment audio signals into the howling suppression model to obtain the howling suppression output of the segment audio signals; randomly generating gain parameters and delay parameters; inputting the howling suppression output, the impulse response and the gain parameters into a howling data generation module to obtain segment howling signals; introducing a delay for the segment howling signal according to the delay parameter, and adding the delayed segment howling signal to the segment audio signal in the clean audio at the same position as the delayed segment howling signal.

[0011] In an optional manner, the howling detection pre-processing is performed by a howling detection pre-processing module, the howling detection pre-processing module and the howling detection network constitute a howling detection model, and the processing process of the sample data of the howling detection model includes: obtaining clean audio and impulse response; generating a random parameter set of each sub-block of the clean audio by block, wherein the random parameter set includes a howling introduction random parameter, a gain parameter and a delay parameter; initializing the howling audio; segmenting the clean audio to obtain a segment audio signal; if the howling suppression is determined according to the howling introduction random parameter of the sub-block to which the segment audio signal belongs, The howling suppression model is used to process the segment audio signal, the segment audio signal is input into the howling suppression model to obtain the howling suppression output of the segment audio signal, and the howling suppression output is used to replace the segment audio signal; the replaced segment audio signal, the impulse response and the gain parameter of the sub-block to which the segment audio signal belongs are input into the howling data generation module to obtain a segment howling signal; a delay is introduced into the segment howling signal according to the delay parameter of the sub-block to which the segment audio signal belongs, and the delayed segment howling signal is added to the segment audio signal in the howling audio at the same position as the delayed segment howling signal.

[0012] In an optional manner, the processing process of the sample data of the howling detection model further includes: adding the delayed segment howling signal to a segment audio signal in the clean audio at the same position as the delayed segment howling signal.

[0013] In an optional manner, the processing process of the sample data of the howling detection model also includes: dividing the howling audio into frames to obtain a frame howling signal; determining the frame energy of the frame howling signal; counting the number of howling sampling points of the frame howling signal according to the frame energy; marking the frame label of the frame howling signal according to the sampling point threshold and the number of howling sampling points; and splicing the frame labels of each frame howling signal in the howling audio to obtain the howling label of the howling audio.

[0014] In an optional manner, the processing flow of the howling data generation module includes: calculating the convolution of the input audio signal and the impulse response to obtain a convolution output; normalizing the convolution output and multiplying it by a gain parameter to obtain a processed convolution output; and limiting the signal boundary value of the processed convolution output to obtain a segment howling signal.

[0015] In the embodiment of the present application, a howling suppression network is used to perform howling suppression processing on audio data to obtain a howling suppression mask, a howling detection network is used to perform howling detection on the audio data to obtain a howling detection output, and the howling detection output of the howling detection model is used to perform weighted optimization on the howling suppression mask. The optimized howling suppression mask can accurately perform howling suppression on the audio data, thereby avoiding the problem of audio distortion in the howling suppression output.

[0016] According to another aspect of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the howling suppression method as described above.

[0017] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the howling suppression method as described above is implemented.

[0018] The electronic device and computer-readable storage medium provided in the present application have the same beneficial effects as the above-mentioned howling suppression method, which will not be described in detail here.

[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0021] Figure 1A schematic diagram of an application scenario provided for an embodiment of the present application;

[0022] Figure 2 A flow chart of constructing and training a howling suppression model provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the structure of a howling suppression network provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the processing flow of the howling data generation module provided in an embodiment of the present application;

[0025] Figure 5 A flow chart of a method for constructing howling detection data provided in an embodiment of the present application;

[0026] Figure 6 A flowchart of the construction and training of a howling detection model provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of a howling detection network provided in an embodiment of the present application;

[0028] Figure 8 A flowchart of a howling suppression method provided in an embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0031] Figure 1 Schematic diagram of application scenarios provided by embodiments of the present application. Camera device 1 establishes a communication connection with terminal device 2 via network 3. Camera device 1 may be a camera for security monitoring, an IP camera, or other video surveillance equipment. Terminal device 2 may be a touch-screen mobile phone, a smart phone, a tablet computer, a portable terminal device, or other terminal electronic device with a display screen. Network 3 includes but is not limited to one or more of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a 4G / 5G network, WIFI, Bluetooth, and a point-to-point (P2P) communication network.

[0032] In the embodiment of the present application, the camera device 1 and the terminal device 2 may include one or more processors, which may be a central processing unit CPU, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the present embodiment, which are not limited here. The one or more processors included in the terminal device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs, which are not limited here.

[0033] The camera device 1 is installed in an area that needs to be monitored (such as a home, office, shopping mall, etc.), so that the camera device 1 can capture monitoring videos in the monitoring area and send the captured videos to the terminal device 2 through the network 3 for users to browse.

[0034] With the development of software and hardware technology, in addition to the basic monitoring function, some camera devices 1 also integrate audio functions, supporting two-way voice intercom through built-in microphones and speakers. Through the application (APP) installed on the terminal device 2, users can have real-time voice calls with people at home or in the office at any time. With the development of artificial intelligence (AI) technology, the camera device 1 integrated with an AI chip, as a smart home device, can interact with the user through voice to achieve multiple functions such as home guarding and video chat.

[0035] In the camera device 1 with voice call and interactive functions, a sound reinforcement system composed of a microphone, a power amplifier and a speaker is usually included. The microphone and the speaker form a closed loop of speaker-power amplifier-speaker, and certain frequency points are continuously strengthened, thereby causing howling. When howling occurs, the speaker will be damaged, hearing will be damaged, etc., affecting the user experience.

[0036] In order to solve the above problems, the prior art generally uses a howling suppression algorithm to suppress acoustic feedback. Commonly used howling suppression algorithms include frequency shifting, phase shifting, notch filters, and adaptive howling suppression algorithms. Among them, the frequency shifting and phase shifting howling suppression algorithms weaken or eliminate the howling sound by introducing a frequency or phase shift in the feedback path. The notch filter method suppresses this frequency component by setting a narrowband filter corresponding to the howling frequency in the audio signal, thereby achieving the purpose of suppressing howling. The adaptive howling suppression algorithm suppresses the howling sound by dynamically adjusting the filter parameters based on the feedback signal monitored in real time. The effect of the adaptive howling suppression algorithm is highly dependent on the tuning of parameters, and the selection of parameters is difficult, especially in complex howling and noise environments. It is more difficult to select appropriate parameters. The generalization ability of the algorithm is also limited. The model is trained based on specific or limited howling data, and there may be overfitting problems, resulting in insufficient generalization ability of the model and poor performance for unknown howling scenes.

[0037] With the development of AI technology, the research or development of howling suppression algorithms based on deep learning has gradually attracted attention. Deep learning is a multi-layer representation learning method that converts the input of the previous layer into an abstract representation with high-level semantic information through nonlinear transformation, thereby learning the complex mapping relationship between input and label. The howling suppression algorithm based on deep learning learns the characteristics of the signal and the statistical laws of howling through a deep neural network model, thereby suppressing the howling signal in the audio.

[0038] Howling suppression algorithms need to perform howling suppression based on howling detection results. However, howling is an occasional event, and the howling suppression effect is closely related to the accuracy of howling detection. Existing howling suppression technologies generally implement howling detection through amplitude and frequency thresholds, which can easily misdetect non-howling signals with high amplitude and high frequency, resulting in audio distortion after howling suppression.

[0039] In addition, the deep neural network model used in the prior art is large in size and complex in structure, and cannot be applied to resource-constrained embedded platforms, and therefore cannot meet the real-time requirements of audio processing. For embedded platforms such as the camera device 1, there are clear restrictions on the size and structure of the neural network model, and the existing howling suppression methods based on deep learning generally cannot meet the requirements of transplantation or real-time performance. In addition, model training relies on a large amount of diverse howling data, and it is difficult to collect rich howling data.

[0040] The embodiment of the present application constructs a howling detection and suppression combined model based on a deep neural network (referred to as the combined model for short), which includes a howling detection model and a howling suppression model. The output of the howling suppression model is weighted and optimized using the continuous prediction probability of the howling detection model to avoid the problem of howling suppression output audio distortion caused by false detection of the howling detection algorithm.

[0041] The embodiments of the present application respectively construct and train a howling suppression model and a howling detection model. Among them, when training the howling suppression model, the situation where there is residual howling after the howling suppression model is processed is used to iteratively generate a segment howling signal, and a delay is introduced to the segment howling signal, and the delayed segment howling signal is added to the clean audio to obtain a training sample that introduces howling, which solves the problem of collecting rich samples, thereby iteratively training the howling suppression model. When training the howling detection model, the pre-trained howling suppression model is randomly used to process the clean audio, thereby simulating the situation where there is residual howling after the howling suppression model is processed, so as to obtain a training sample that randomly introduces howling, which not only solves the problem of sample collection, but also the random howling samples can improve the generalization ability of the howling detection model, thereby improving the detection accuracy of the howling detection model. The construction and training process of the howling suppression model and the howling detection model are respectively introduced below.

[0042] First, the construction and training process of the howling suppression model is introduced. Figure 2 The following is a flowchart of the construction and training of the howling suppression model provided in the embodiment of the present application. The method can be executed by electronic devices such as computers and servers. Figure 2 Describe the construction and training process of the howling suppression model.

[0043] S201, constructing a howling suppression dataset.

[0044] The howling suppression model of the embodiment of the present application is obtained based on deep neural network training. Before training the deep neural network, the sample data required for training is first constructed. This step collects clean audio and room pulse response (i.e., impulse response) data, which can clean the interference signals in the clean audio and impulse response data, and cut the clean audio into segments of fixed length (e.g., 15s). The audio segments and impulse responses are randomly divided into training groups, verification groups, and test groups in the ratio of a:b:c (e.g., 12:3:1) to obtain a howling suppression training data set, a howling suppression verification data set, and a howling suppression test data set. Among them, the training group is used to learn model parameters, the verification group is used to adjust the model configuration and prevent overfitting, and the test group is used to evaluate the final performance of the model. The above division of the data set can ensure that the model not only performs well on the training data, but also can be generalized to unseen data.

[0045] The clean audio is segmented so that the audio data is segmented and input into the howling suppression model for model training. The clean audio in the howling suppression dataset can be used to generate the initial input audio segment when training the howling suppression model. The segmented howling signal is also added to the original audio to iteratively generate the segmented audio signal for model training, and is used as the label of the segmented audio signal.

[0046] The clean audio and impulse response obtained in this step can be from open source data, or can be acquired by the camera device 1, for example, by acquiring video data of the camera device 1 in a quiet environment, and separating the original background noise data of the camera device 1 to obtain clean audio. The clean audio in the embodiment of the present application is not limited to human voice, but can also include alarm sounds, music sounds, bird sounds, insect sounds, rain sounds, and traffic sounds.

[0047] S202, constructing a howling suppression pre-processing module.

[0048] The clean audio in the howling suppression data set is input into the howling suppression pre-processing module for pre-processing. The processing flow of the constructed howling suppression pre-processing module on the audio data is as follows:

[0049] First, the audio data (i.e., clean audio) is framed according to a fixed size and offset, and a window function is added frame by frame. Before performing time-frequency conversion on the audio data, the audio data can be framed according to a fixed size (e.g., Fourier transform size) and an offset. Framing can divide a long time domain signal into multiple short time domain frames, making it easier to process each frame independently, and more effectively analyze the characteristics of the audio signal, extract useful features from it, and improve processing efficiency. After framing, window functions (such as Hanning window and Hamming window) are added to the audio data frame by frame to achieve signal smoothing processing to reduce the amplitude of the overlapping parts between frames, thereby reducing audio mutations.

[0050] Then, the audio data is transferred from the time domain to the frequency domain frame by frame to obtain frequency domain audio. Since the audio signal is represented in the frequency domain, its frequency components can be more easily analyzed, which is crucial for understanding the characteristics of howling. In the frequency domain, specific frequency components can be more effectively separated or extracted to separate clean audio and howling signals, and the time-varying characteristics of the signal can also be analyzed in the frequency domain. Therefore, this step converts the audio data from the time domain to the frequency domain. Converting audio data from the time domain to the frequency domain can be achieved through Fourier transform. The audio data converted to the frequency domain can extract the amplitude and phase information of each frame.

[0051] Finally, the amplitude of each frame in the frequency domain audio is spliced ​​and the phase of each frame is spliced ​​to generate splicing features. The size of the amplitude feature and phase feature of each frame in the frequency domain audio is [data batch, amplitude / phase, frame length, number of frames]. The feature size after splicing and merging is [data batch, 2, frame length, number of frames], where 2 represents the amplitude channel and phase channel. The frame length is the information of the frequency dimension, which represents the number of frequency sampling points in the frequency domain and is related to the size of the Fourier transform. The number of frames is the information of the time dimension, and the number of frames for each frame is 1.

[0052] The dimensions of the obtained spliced ​​features include data batches, feature channels, frame lengths, and number of frames. Feature channels include amplitude channels and phase channels. That is, after splicing the information of each frame, a four-dimensional feature is obtained, and the four dimensions represent the training data batch (if it is a verification batch during verification, it is a test batch during testing), feature channels (amplitude and phase), frame length, and number of frames. The dimensions of the training / verification / test batch are determined by step S110, which affects the model training and convergence speed; the feature channel, frame length, and frame number dimensional features will be learned by the deep neural network to extract relevant feature representations, where the feature channel is formed after splicing, and the frame length and number of frames are obtained after Fourier transform.

[0053] When the amplitude of each frame in the frequency domain audio is spliced ​​and the phase of each frame is spliced, it specifically includes: splicing the amplitude of each frame in the frequency domain audio in two dimensions of frame length and frame number to obtain an amplitude feature map, wherein the length of the amplitude feature map is the frame length, and the width of the amplitude feature map is the frame number; splicing the phase of each frame in the frequency domain audio in two dimensions of frame length and frame number to obtain a phase feature map, wherein the length of the phase feature map is the frame length, and the width of the phase feature map is the frame number; and channel splicing the amplitude feature map and the phase feature map, and adding data batches to the features after channel splicing to generate splicing features.

[0054] The spliced ​​features obtained by the splicing operation contain two feature channels: the amplitude channel and the phase channel. In the deep neural network, the two-dimensional convolution is combined to perform feature extraction on the two feature channels of amplitude and phase, which is convenient for fitting the corresponding amplitude and phase respectively, simplifies the structure of the trained howling suppression model, compresses the size of the howling suppression model, and makes the howling suppression model easier to integrate on the embedded end.

[0055] The splicing features include four dimensions: data batch, feature channel, frame length and number of frames. These four dimensions are used as separate dimensions, among which the feature channel, frame length and number of frames are used for feature learning of deep neural network to improve the howling suppression effect. The frame length and number of frames are used as the length and width of the amplitude feature map and the phase feature map respectively, so that the deep neural network can simultaneously learn the feature representation of the two dimensions of frame length (frequency domain) and frame number (time domain), which not only improves the learning efficiency, but also ensures the performance of the howling suppression model.

[0056] The raw data is converted into a format suitable for input into the howling suppression network through pre-processing, so that the howling suppression network can learn and generalize more effectively.

[0057] In some embodiments, the concatenated features may also be normalized using a tanh function. By performing Tanh normalization on the output of the Fourier transform, the feature value range may be compressed, model training may be accelerated, and quantization processing of the embedded platform may be facilitated.

[0058] S203: Building a howling suppression network.

[0059] The spliced ​​features after pre-processing by the howling suppression pre-processing module are input into the howling suppression network for howling suppression. The spliced ​​features are referred to as the first input audio in the following text.

[0060] The howling suppression network includes an encoder module, a recurrent neural network layer and a decoder module. The encoder module uses two-dimensional convolution to perform sliding window operations on the amplitude channel and the phase channel in both the frame length and the number of frames, so as to extract amplitude features and phase features respectively, and obtain the first feature. The recurrent neural network layer extracts the first feature according to the temporal context information of the first feature to obtain the second feature. The decoder module uses two-dimensional transposed convolution to perform sliding window operations on the amplitude channel and the phase channel in both the frame length and the number of frames, so as to reconstruct the amplitude features and the phase features respectively, and obtain the howling suppression mask.

[0061] Figure 3 This is a schematic diagram of the structure of the howling suppression network provided in the embodiment of the present application, in which (a) is a schematic diagram of the structure of the howling suppression network, (b) is a schematic diagram of the structure of the encoder in the howling suppression network, and (c) is a schematic diagram of the structure of the decoder in the howling suppression network. Please refer to Figure 3 In an embodiment of the present application, the encoder module includes a first encoder, a second encoder, and a third encoder that are cascaded in sequence, and each encoder includes a first convolutional layer, a first regularization layer, a first activation layer, a second convolutional layer, a second regularization layer, and a second activation layer that are cascaded in sequence; the decoder module includes a first decoder, a second decoder, and a third decoder that are cascaded in sequence, and each decoder includes a first transposed convolutional layer, a third regularization layer, a third activation layer, a second transposed convolutional layer, a fourth regularization layer, and a fourth activation layer that are cascaded in sequence.

[0062] As shown in the figure, the encoder module includes 18 layers, and the structure is: convolution layer 1->regularization layer 1->activation layer 1->convolution layer 2->regularization layer 2->activation layer 2->convolution layer 3->regularization layer 3->activation layer 3->convolution layer 4->regularization layer 4->activation layer 4->convolution layer 5->regularization layer 5->activation layer 5->convolution layer 6->regularization layer 6->activation layer 6.

[0063] The number of convolution kernels in convolution layer 1 to convolution layer 6 can be set to 16, 16, 32, 32, 64, 64 respectively, and the convolution kernel sizes are set to 5x1, 3x1, 3x1, 3x1, 3x1, 3x1 respectively, and the step sizes are set to 2x1. The activation layers all use Rectified Linear Unit (ReLU) as the activation function, and the regularization layers all use Batch Normalization (BN).

[0064] The recurrent neural network layer can use a long short-term memory network (LSTM), such as a 1-layer LSTM, with a hidden layer size of 96. The long short-term memory network can learn long-term dependencies and avoid the gradient disappearance and gradient explosion caused by the traditional recurrent neural network (RNN), and has good generalization performance.

[0065] The decoder module consists of 18 layers, and the structure is as follows: transposed convolution layer 1->regularization layer 7->activation layer 7->transposed convolution layer 2->regularization layer 8->activation layer 8->transposed convolution layer 3->regularization layer 9->activation layer 9->transposed convolution layer 4->regularization layer 10->activation layer 10->transposed convolution layer 5->regularization layer 11->activation layer 11->transposed convolution layer 6->regularization layer 12->activation layer 12.

[0066] The number of convolution kernels in transposed convolution layer 1 to transposed convolution layer 6 can be set to 64, 32, 32, 16, 16, 2, respectively, and the convolution kernel sizes are set to 3x1, 3x1, 3x1, 3x1, 3x1, 5x1, respectively, and the step sizes are all set to 2x1. The number of convolution kernels in convolution layer 7 is set to 2, the convolution kernel size is set to 1x1, and the step size is set to 1. Activation layers 7 to 12 all use ReLU activation function, and regularization layers all use BN normalization method.

[0067] After howling suppression is performed by the howling suppression network, a howling suppression mask is obtained. The howling suppression mask is the output of the activation layer, indicating the strength of howling suppression, which is a continuous value from 0 to 1. 0 means full suppression, and 1 means no suppression.

[0068] S204, constructing a howling suppression post-processing module.

[0069] The howling suppression mask obtained after howling suppression by the howling suppression network is input into the howling suppression post-processing module for post-processing.

[0070] In an embodiment of the present application, the howling suppression post-processing module processes the howling suppression mask weighted by the howling detection model. When the howling suppression post-processing module processes the weighted howling suppression mask, the weighted howling suppression mask is first multiplied element by element with the first input audio to obtain the amplitude and phase of each frame after howling suppression. The weighted howling suppression mask can improve the performance and generalization ability of the model by selectively focusing on or ignoring different parts of the data. By multiplying the weighted howling suppression mask with the first input audio containing howling, the sound that is desired to be retained can be enhanced while suppressing unwanted howling.

[0071] Then, the amplitude and phase of each frame after howling suppression are converted from the frequency domain to the time domain frame by frame, and the time domain data of each frame after howling suppression is obtained. Since the final output howling suppression audio needs to be a time domain signal, this step performs the frequency domain to time domain conversion, which can be achieved by inverse Fourier transform, and a Hanning window can be added.

[0072] Finally, the overlapping parts of adjacent frames in the time domain data of each frame after howling suppression are weighted averaged to obtain howling suppressed audio that is continuous in the time domain.

[0073] This step smoothly superimposes the waveforms of the overlapping parts to reduce the sudden changes in the audio. The resulting howling suppression audio is a continuous time domain output signal, that is, the output of the howling suppression network is converted into a continuous time domain output signal required by the end user or downstream tasks.

[0074] S205, defining a howling suppression target loss function.

[0075] The target loss function is used to measure the difference between the model prediction result and the true label. The loss value of the loss function is minimized during the training process so that the model can better fit the training data and improve the generalization ability of unseen data. snr Loss, i.e. prediction result The signal-to-noise ratio (SNR) loss of the label S (i.e., howling suppressed audio) and the label S (i.e., clean audio) is calculated as follows:

[0076]

[0077] The square sum ratio of the clean audio and the howling signal (ie, noise) is calculated by formula 1, and the negative value of the logarithmic operation is taken as the loss result. Formula 1 processes the time domain signal that has not been pre-processed.

[0078] S206, randomly selecting a fixed number of samples from the howling suppression training data set and the howling suppression verification data set to form a training group and a verification group of data for the current training round, each sample including clean audio and impulse response data.

[0079] S207, copy the clean audio from the training group as a label, generate silent audio of the same size as the clean audio as the initial model output, and segment the clean audio according to a fixed size and offset to obtain a segment audio signal.

[0080] This step generates the labels required for howling suppression model training, initializes the output of the howling suppression model, and generates the segmented audio signal for input into the howling suppression model. The howling suppression model processes the segmented clean audio. Segmenting the clean audio into fixed-size segments can simulate the audio data buffer setting of the embedded platform, thereby better adapting to the embedded platform and being able to iteratively generate howling data based on the segmented audio signal.

[0081] S208: Update the audio sub-segment to the next audio signal segment.

[0082] When this step is first executed, the first audio signal is used as an audio sub-segment. When this step is subsequently executed in a loop, the second audio signal, the third audio signal, etc. are updated as audio sub-segments in sequence, so as to perform howling suppression processing on each audio signal.

[0083] S209, the audio sub-segment is processed in sequence by a howling suppression pre-processing module, a howling suppression network and a howling suppression post-processing module to obtain a howling suppression output of the audio sub-segment.

[0084] S210: Use the howling suppression output of the audio sub-segment to update the howling suppression model output, and randomly generate a gain parameter and a delay parameter.

[0085] Wherein, using the howling suppression output of the audio sub-segment to update the howling suppression model output means replacing the howling suppression output of the current audio sub-segment with the corresponding sub-segment in the initial model output in S207. For example, if the audio sub-segment updated in S208 is the first audio signal, then in this step, the howling suppression output of the current audio sub-segment replaces the first output in the initial model output; if the audio sub-segment updated in S208 is the second audio signal, then in this step, the howling suppression output of the current audio sub-segment replaces the second output in the initial model output... and so on.

[0086] The value range of the gain parameter is a floating point number between 1 and 3, for example. The value range of the delay parameter is 200 to 2500ms. During the loop processing, the gain parameter and delay parameter generated for each audio sub-segment are random values, which can increase the diversity of training data and improve the generalization ability of the model, thereby reducing the risk of overfitting.

[0087] S211, using the howling suppression model output, impulse response and gain parameter as input of the howling data generation module, generating a segment howling signal, introducing a delay for the segment howling signal according to the delay parameter, and adding the delayed segment howling signal to the clean audio.

[0088] The howling data generating module is equivalent to an analog loudspeaker, and a segment howling signal can be obtained through the howling data generating module. Figure 4Schematic diagram of the processing flow of the howling data generation module provided in the embodiment of the present application. Figure 4 As shown, the processing flow of the howling data generation module includes the following steps:

[0089] S401, calculating the convolution of the segment audio signal and the impulse response.

[0090] In the embodiment of the present application, the output of the howling suppression model obtained in S210 is used as the input segment audio signal of the howling data generation module, that is, the output of the howling suppression model is used as the speaker back-sampled analog signal, and the convolution of the segment audio signal and the impulse response is calculated. The impulse response is the impulse response data in the sample to which the segment audio signal selected in S206 belongs.

[0091] S402, normalize the convolution output and multiply it by a gain parameter.

[0092] This step multiplies the normalized convolution output by a gain parameter to simulate a power amplifier in a speaker system.

[0093] S403, performing signal boundary value limitation to obtain a segment howling signal.

[0094] The calculation formulas for S401 to S403 are as follows:

[0095]

[0096] Among them, s and e represent the segment audio signal and convolution output respectively, and M represents the number of sample points. First, the root mean square feature (Root Mean Square, RMS) s_rms of the segment audio signal s and the root mean square feature e_rms of the convolution output e are calculated by Formula 2 and Formula 3 respectively. Then, the processed signal s_norm is obtained by normalization processing through Formula 4 and multiplied by the gain parameter gain. Finally, the boundary value of s_norm is limited by Formula 5, and the normalized signal s_norm_clip is output. The normalized signal s_norm_clip is the segment howling signal.

[0097] After the howling data generation module generates the segment howling signal, a delay is introduced into the segment howling signal according to the delay parameter, and the delayed segment howling signal is added to the clean audio. When the delayed segment howling signal is added to the clean audio, for example, if the delay parameter is 200ms, the delayed segment howling signal is added to the position of the current audio subsegment in the clean audio that is delayed by 200ms. The audio subsegment processed in S209 includes the original audio subsegment plus the added segment howling signal (if the segment has a segment howling signal added).

[0098] A segment howling signal is generated through a howling data generation module, and the delayed segment howling signal is added to the clean audio, so as to iteratively generate howling data to automatically generate training samples, which can reduce the labor and time costs of collecting howling data in actual scenarios.

[0099] S212, determine whether the audio sub-segment is the last audio signal, if so, execute S212, if not, go to S208 and continue to process the next audio segment.

[0100] If the currently processed audio sub-segment is not the last audio signal, it means that there are still unprocessed audio signals in the current sample, then go to S208 to continue processing the next audio segment. If the currently processed audio sub-segment is the last audio signal, it means that the howling suppression processing has been completed for each audio signal in the current sample, and the howling suppression model output can be used to calculate the loss function to optimize the model.

[0101] S213, using the howling suppression model output and the label as inputs of the howling suppression target loss function, calculating the loss value, using the back propagation algorithm to calculate the gradient of the howling suppression loss function to the howling suppression network parameters, and updating the howling suppression network parameters according to the gradient.

[0102] This step uses the howling suppression target loss function defined in S205 to calculate the loss value, wherein the howling suppression model output is the howling suppression model output updated by S209, which is the complete howling suppression output obtained by splicing, including each howling suppression output corresponding to the segmented audio signal.

[0103] Back propagation refers to calculating the gradient value of each neuron starting from the output layer. First, the gradient of the output layer is calculated, and then the gradient of each hidden layer is calculated layer by layer until the input layer is reached. The calculation of the gradient is based on the chain rule, which multiplies the gradient of the previous layer by the derivative of the activation function of the current layer.

[0104] Gradient update uses the calculated gradient values ​​to update the network parameters, including the weights and biases of the convolutional layer, transposed convolutional layer, and regularization layer.

[0105] S214, using the validation group data of the current training round to calculate the validation group loss value of the current training round.

[0106] In this step, the validation group loss value of the current training round (referred to as "current loss") can be obtained by passing the validation group data of the current training round through S207 to S212.

[0107] S215, determine whether the loss is reduced, if so, execute S216, if not, execute S217.

[0108] S216, save the current network weight of the howling suppression network, and update the optimal loss to the current loss.

[0109] In S215, the current loss is compared with the optimal loss (the training is initially set to a maximum constant value). If the current loss is less than the optimal loss, the current network weight of the howling suppression network is saved in S216, and the optimal loss is updated to the current loss.

[0110] S217, determine whether the cumulative round loss is reduced, if so, end the process, if not, execute S218.

[0111] If the validation set loss value no longer decreases within a specified round, which means that the performance on the validation set no longer improves, the training can be terminated early. This helps to stop training when the model performance no longer improves and prevent waste of time and resources.

[0112] S218, determine whether the maximum number of training rounds has been reached. If so, end the process. If not, go to S206. If not, return to S206 to continue selecting the next round of training group and validation group data for training. In order to prevent overfitting, save computing resources, control training time, and balance accuracy and efficiency, the maximum number of training rounds can be set. If the set maximum number of training rounds is reached, end the process to obtain a trained model for deployment.

[0113] Steps S215 to S218 compare the current loss of the validation group in the current training round with the optimal loss. If the current target loss is less than the optimal loss, the current deep neural network weight is saved and the optimal loss is updated. The training stops until the current loss of the validation group no longer decreases within the specified round or the training reaches the set maximum round, and a trained howling suppression model is obtained. That is, a trained howling suppression model is obtained after the process ends at S217 or S218.

[0114] Next, the construction and training process of the howling detection model are introduced. First, the construction of the howling detection data used to train the howling detection model is introduced. Figure 5 A flow chart of a method for constructing howling detection data provided in an embodiment of the present application.

[0115] S501, randomly selecting clean audio and impulse response data from a howling suppression training data set.

[0116] S502: randomly divide the clean audio into blocks and generate a random parameter set for each sub-block.

[0117] In this step, the selected clean audio is first randomly divided into multiple sub-blocks, such as 1, 2, 3 or 4 blocks, and the starting position of each sub-block is marked. Then, each sub-block randomly selects whether to introduce howling. If howling is introduced, it randomly selects whether to use the howling suppression model (i.e., after howling is introduced) for this sub-block. Figure 2 The howling suppression model obtained by training in the embodiment shown, or the howling suppression model that has been pre-trained for several rounds) is processed to simulate the situation where there is residual howling after the howling suppression model is processed. In addition, for each sub-block, the speaker and the power amplifier are simulated to randomly generate gain parameters, and the delay parameters are randomly generated by simulating the time consumption of the sound path of the howling scene.

[0118] The random parameter of whether to introduce howling can indicate whether to introduce howling by taking a value, for example, including two values ​​of 0 and 1. A value of 0 indicates no howling is introduced, and a value of 1 indicates howling is introduced. The same is true for the random parameter of whether to use the howling suppression model for processing.

[0119] In summary, the random parameter set includes a random parameter for whether to introduce howling, a random parameter for whether to use a howling suppression model, a gain parameter, and a delay parameter. The random parameter for whether to introduce howling and the random parameter for whether to use a howling suppression model belong to the howling introduction random parameter, and the two together determine whether to use the howling suppression model to process each sub-segment belonging to the sub-block.

[0120] By dividing the clean audio into blocks and introducing different random parameters, the data richness is improved, the diversity of training data is increased, and the generalization ability of the model is improved, thereby reducing the risk of overfitting.

[0121] S503: Generate silent audio of the same size as the clean audio as the initial howling audio, and segment the clean audio according to a fixed size and offset to obtain segment audio signals.

[0122] The size of the generated silent audio is the same as the size of the unblocked clean audio, so the size of the initial howling audio is the same as the size of the unblocked clean audio. Segmenting the clean audio according to a fixed size can simulate the audio data buffer setting of the embedded platform, thereby better adapting to the embedded platform and being able to iteratively generate howling data based on the segment audio signal.

[0123] S504: Generate a silence segment of the same size as the segment audio signal as an initial segment howling signal.

[0124] This step uses the silent segment to assign an initial value to the segment howling signal.

[0125] S505: Update the sub-segment to the next audio signal.

[0126] When this step is first executed, the first audio signal (i.e., the starting segment in the clean audio) is used as the sub-segment (i.e., the current audio signal). When this step is subsequently executed in a loop, the second audio signal, the third audio signal, etc. are updated as sub-segments in sequence.

[0127] S506, determine whether the sub-segment belongs to the current sub-block. If yes, execute S508, if not, execute S507.

[0128] S507, update the random parameter set to the parameters of the next sub-block, and then execute S508.

[0129] The embodiment of the present application processes each segment of the audio signal in sequence in a sliding window manner, and each segment of the audio signal may belong to the first sub-block, the second sub-block or other sub-blocks. And each sub-block has a different set of random parameters. S505 can determine whether the sub-segment number belongs to the current sub-block according to the starting position of the sub-block. If the sub-segment does not belong to the current sub-block, it means that the segment audio signals belonging to the current sub-block have been processed by the model, and the sub-segment is the first sub-segment of the next sub-block. The sub-segment cannot use the random parameter set of the current sub-block, and the random parameter set needs to be updated to the random parameter set of the next sub-block.

[0130] S508, judging whether howling is introduced according to the random parameter set, if so, executing S509, if not, turning to S505.

[0131] If the sub-segment belongs to the current sub-block, there is no need to update the random parameter set, and the judgment of whether to introduce howling and whether to use the howling suppression model is directly based on the random parameter set of the current sub-block. If the random parameter set does not introduce howling, the processing of the current sub-segment is terminated and the sub-segment is updated to the next audio signal. For example, if the random parameter value of whether to introduce howling is 0, it is judged that howling is not introduced; if the value is 1, it is judged that howling is introduced.

[0132] S509: Determine whether to use the howling suppression model for processing according to the random parameter set. If yes, execute S510; if no, execute S511.

[0133] For example, if the random parameter of whether to use the howling suppression model processing takes a value of 0, it is determined that the howling suppression model processing is not used; if the random parameter takes a value of 1, it is determined that the howling suppression model processing is used.

[0134] S510: Input the sub-segment into a howling suppression model to obtain a howling suppression output, and use the howling suppression output to replace the sub-segment.

[0135] After the sub-segment is input into the howling suppression model, it is processed in turn by the howling suppression pre-processing module, the howling suppression network and the howling suppression post-processing module to obtain the howling suppression output, and then the howling suppression output is used to replace the sub-segment (i.e., the audio signal).

[0136] S511, input the sub-segment, impulse response and gain parameter into a howling data generation module to obtain a segment howling signal.

[0137] If it is determined according to the random parameter set that the howling suppression model is not used for processing, then in this step, the sub-segment that is not processed by the howling suppression model is input into the howling data generation module; if it is determined according to the random parameter set that the howling suppression model is used for processing, then in this step, the sub-segment processed by the howling suppression model in S510 is input into the howling data generation module as a segment audio signal. The processing flow of the howling data generation module is described above and will not be repeated here.

[0138] S512, introducing a delay for the segment howling signal according to the delay parameter, and adding the delayed segment howling signal to the howling audio.

[0139] Adding the delayed segment howling signal to the howling audio can record the howling and generate a frame label. The initial howling audio is the silent audio generated in S503, and the delayed segment howling signal is added to iteratively update the howling audio, so as to obtain a complete howling label based on the frame label of each frame.

[0140] S513, determine whether the sub-segment is the last audio signal, if so, execute S515, if not, execute S514.

[0141] S514, adding the delayed segment howling signal to the clean audio, and then going to S505.

[0142] If the sub-segment is not the last audio signal, adding the delayed howling signal to the clean audio can simulate the howling generation mechanism (that is, the signal is iteratively transmitted in the closed loop between the speaker-power amplifier-microphone), and then updating the sub-segment to the next audio signal, and continuing to process the next audio signal.

[0143] S515, dividing the howling audio into frames according to a fixed size and offset.

[0144] If the sub-segment is the last audio signal, it means that the segment howling signal of the sample has been iteratively generated segment by segment, and the sample can be framed and labeled. In this step, the frame howling signal can be obtained by framing the Fourier transform size and offset, and the energy threshold and sampling point threshold of the frame howling signal can be set.

[0145] S516, calculating the frame energy of the frame howling signal, and counting the number of howling sampling points.

[0146] In this step, the energy sequence of the frame howling signal (current frame) can be calculated by squaring, and the total sampling point data exceeding the energy threshold in the frame howling signal can be counted.

[0147] S517, marking a frame label of the frame howling signal.

[0148] This step marks the frame label according to the sampling point threshold. When the total sampling point data exceeds the sampling point threshold, the frame is determined to be a howling frame and its frame label is marked as 1, otherwise, its howling label is marked as 0.

[0149] S518, determining whether the current frame howling signal is the last frame, if so, executing S519, if not, executing S520.

[0150] S519: Concatenate the frame labels of the frames to obtain a howling label of the howling audio.

[0151] If the howling signal of the current frame is the last frame, the frame labels of each frame are concatenated to obtain a howling label sequence consisting of 0 and 1, and the sequence size is the number of frames. The original clean audio and howling label sequence are saved.

[0152] S520, update the current frame to the next frame of howling signal, and go to S516.

[0153] If the current frame howling signal is not the last frame, the current frame is updated to the next frame howling signal, and labeling processing of the next frame howling signal is performed.

[0154] Through the above method, the howling audio as sample data and the howling label as the label of the sample data are obtained. The above steps S501 to S520 can be performed multiple times to sequentially construct a howling detection training data set, a howling detection verification data set and a howling detection test data set.

[0155] The embodiment of the present application randomly uses a pre-trained howling suppression model to process clean audio, thereby simulating the situation where there is residual howling after the howling suppression model is processed, and generates a segment howling signal through a howling data generation module, and adds the delayed segment howling signal to the clean audio, thereby iteratively generating howling data to automatically generate training samples, which can reduce the labor and time costs of collecting howling data in actual scenarios. By randomly introducing howling through a random parameter set, the generalization ability of the howling detection model can also be improved, and the detection accuracy of the howling detection model can be improved.

[0156] Figure 6 The following is a flowchart of the construction and training of the howling detection model provided in the embodiment of the present application. The method can be executed by electronic devices such as computers and servers. Figure 6 Describe the construction and training process of the howling detection model.

[0157] S601, constructing a howling detection data set.

[0158] This step can be done by Figure 5 The howling detection data set is constructed in the manner of the illustrated embodiment, including a howling detection training data set, a howling detection verification data set, and a howling detection test data set.

[0159] S602, constructing a howling detection pre-processing module.

[0160] The clean audio in the howling detection data set is input into the howling detection pre-processing module for pre-processing. The processing flow of the constructed howling detection pre-processing module for audio data is as follows:

[0161] First, the audio data (ie, clean audio) is segmented according to a fixed size and offset to obtain segmented audio signals. Segmenting the audio data according to a fixed size can simulate the audio data buffer setting of the embedded platform.

[0162] Then, each audio signal is framed according to a fixed size and offset, and a window function is added frame by frame. Framing can be performed through Fourier transform. Framing can divide the long time domain signal into multiple short time domain frames, which is convenient for independent processing of each frame, and more effectively analyzes the characteristics of the audio signal, extracts useful features from it, and improves processing efficiency. After framing, window functions (such as Hanning window and Hamming window) are added to the audio data frame by frame to achieve signal smoothing processing to reduce the amplitude of the overlapping part between frames, thereby reducing audio mutations.

[0163] Next, the audio data is transferred from the time domain to the frequency domain frame by frame to obtain frequency domain audio. Since the audio signal is represented in the frequency domain, its frequency components can be more easily analyzed, which is crucial for understanding the characteristics of howling. In the frequency domain, specific frequency components can be more effectively separated or extracted to separate clean audio and howling signals, and the time-varying characteristics of the signal can also be analyzed in the frequency domain. Therefore, this step converts the audio data from the time domain to the frequency domain. Converting audio data from the time domain to the frequency domain can be achieved through Fourier transform. The audio data converted to the frequency domain can extract the amplitude and phase information of each frame.

[0164] The energy spectrum, Mel spectrum and logarithmic Mel spectrum of each frame are calculated in turn. The calculation formula is as follows:

[0165] pow_spec=mag * mag Formula 6

[0166] mel_spec=melspectrogram(pow_spec,n_mel) Formula 7

[0167] mel_spec_db=10*log 10 mel_spec Formula 8

[0168] Among them, mag represents the frame amplitude, and the square is obtained to obtain the frame energy spectrum pow_spec. n_mel represents the number of mel filters (that is, the frame length). The frame energy spectrum pow_spec is calculated by the mel spectrum conversion function melspectrogram to obtain the frame mel spectrum mel_spec. Finally, the frame mel spectrum mel_spec is calculated by logarithm to obtain the logarithmic mel spectrum mel_spec_db.

[0169] The logarithmic Mel spectrum of each frame is spliced ​​into a four-dimensional feature as the input of the howling detection model. The four dimensions represent the training data batch (if it is a verification batch during verification, it is a test batch during testing), the feature channel (1, indicating only amplitude features), the frame length (n_mel) and the number of frames. The dimensions of the training / verification / test batch are determined by step S601, which affects the model training and convergence speed; the feature channel, frame length and frame number dimensional features will be learned by the deep neural network to extract relevant feature representations, where the feature channel is formed after splicing, and the frame length and frame number are obtained after framing each audio signal according to a fixed size and offset (such as Fourier transform).

[0170] S603: Building a howling detection network.

[0171] The spliced ​​features after pre-processing by the howling detection pre-processing module are input into the howling detection network for howling detection. The spliced ​​features are referred to as the second input audio in the following text.

[0172] The howling detection network includes a feature extraction module, a recurrent neural network layer, and a detection output module. The feature extraction module takes the second input audio as input, and uses two-dimensional convolution to perform sliding window operations in both frame length and frame number dimensions to obtain the third feature. The recurrent neural network layer extracts the third feature based on the temporal context information of the third feature to obtain the fourth feature. The detection output module uses a fully connected layer to reduce the dimension of the fourth feature in the channel dimension, and obtains the howling prediction probability value through the activation layer.

[0173] Figure 7 The schematic diagram of the structure of the howling detection network provided in the embodiment of the present application, (a) is a schematic diagram of the structure of the howling detection network, (b) is a schematic diagram of the structure of the encoder in the feature extraction module, and (c) is a schematic diagram of the structure of the detection output module. Figure 7 The feature extraction module includes a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder and a sixth encoder which are cascaded in sequence, and each encoder includes a first convolutional layer, a first regularization layer, a first activation layer, a second convolutional layer, a second regularization layer and a second activation layer which are cascaded in sequence; the detection output module includes a first fully connected layer, a third activation layer, a second fully connected layer and a fourth activation layer which are cascaded in sequence.

[0174] As shown in the figure, the feature extraction module includes 36 layers, and the structure is: convolution layer 1->regularization layer 1->activation layer 1->convolution layer 2->regularization layer 2->activation layer 2->convolution layer 3->regularization layer 3->activation layer 3->convolution layer 4->regularization layer 4->activation layer 4->convolution layer 5->regularization layer 5->activation layer 5->convolution layer 6->regularization layer 6->activation layer 6->convolution layer 7->regularization layer 7->activation layer 7->convolution layer 8->regularization layer 8->activation layer 8->convolution layer 9->regularization layer 9->activation layer 9->convolution layer 10->regularization layer 10->activation layer 10->convolution layer 11->regularization layer 11->activation layer 11->convolution layer 12->regularization layer 12->activation layer 12.

[0175] The number of convolution kernels in convolution layer 1 to convolution layer 12 can be set to 8, 8, 16, 16, 32, 32, 32, 32, 64, 64, 64, 64 respectively and the convolution kernel sizes can be set to 3x1, 1x1, 3x1, 1x1, 3x1, 1x1, 3x1, 1x1, 3x1, 1x1, 3x1, 1x1, and the step sizes can be set to 2x1, 1x1, 2x1, 1x1, 2x1, 1x1, 2x1, 1x1, 2x1, 1x1, 2x1, 1x1, 1x1 respectively. The activation layers all use the ReLU activation function and the regularization layers all use the BN normalization method.

[0176] The recurrent neural network layer can use LSTM, for example, a 2-layer LSTM with a hidden layer size of 64. The long short-term memory network can learn long-term dependencies, avoid gradient vanishing and gradient explosion caused by traditional RNN, and has good generalization performance.

[0177] The detection output module consists of 2 fully connected layers and 2 activation layers, and the structure is: fully connected layer 1->activation layer 13->fully connected layer 2->activation layer 14. The input size of fully connected layer 1 and fully connected layer 2 can be set to 64 and 64 respectively, and the output size can be set to 64 and 1 respectively. Activation layer 13 uses the ReLU activation function, and activation layer 14 uses the sigmoid activation function.

[0178] After howling detection is performed by the howling detection network, the howling prediction probability is obtained.

[0179] S604: Define a howling detection target loss function.

[0180] The target loss function is used to measure the difference between the prediction results of the howling detection model and the true label. The loss value of the target loss function is minimized during the training process so that the model can better fit the training data and improve the generalization ability of unseen data.

[0181] The howling detection task uses binary cross-entropy loss (Binary Cross-Entropy loss, L bce Loss), calculate the binary cross entropy loss of the model output probability value and the true label, the calculation formula is as follows:

[0182]

[0183] Among them, M represents the number of samples, y and Represent the true label and predicted probability value of a single sample respectively.

[0184] S605 , randomly selecting a fixed number of samples from the howling detection training data set and the howling detection verification data set to form a training group and a verification group data of the current training round, each sample including a clean audio and a howling label.

[0185] S606, the clean audio is processed by the howling detection pre-processing module and the howling detection network in sequence to obtain a howling prediction probability sequence including the howling prediction probability of each frame.

[0186] Since the howling detection pre-processing module divides each audio signal into frames according to a fixed size and offset, the obtained second input audio contains the features of each frame. Therefore, the concatenated features containing the logarithmic Mel spectrum of each frame are input into the howling detection network to obtain the howling prediction probability of each frame.

[0187] S607, using the howling prediction probability sequence and the howling label as inputs of the howling detection target loss function, calculating the loss value, using the back propagation algorithm to calculate the gradient of the howling detection loss function to the deep neural network parameters, and updating the howling detection network parameters according to the gradient.

[0188] S608, using the validation group data of the current training round to calculate the validation group loss value of the current training round.

[0189] In this step, the validation group loss value of the current training round (referred to as "current loss") can be obtained by passing the validation group data of the current training round through S606 to S607.

[0190] S609, determine whether the loss is reduced, if so, execute S610, if not, execute S611.

[0191] S610, save the current network weight of the howling detection network, and update the optimal loss to the current loss.

[0192] Compare the current loss with the optimal loss (set to a maximum constant value at the beginning of the training). If the current loss is less than the optimal loss, save the current network weights of the howling detection network and update the optimal loss to the current loss.

[0193] S611, determine whether the cumulative round loss is reduced, if so, end the process, if not, execute S612.

[0194] If the validation set loss value no longer decreases within a specified round, which means that the performance on the validation set no longer improves, the training can be terminated early. This helps to stop training when the model performance no longer improves and prevent waste of time and resources.

[0195] S612, determine whether the maximum number of training rounds has been reached, if so, end the process, if not, go to S605.

[0196] If the maximum number of training rounds is not reached, the process returns to S605 to continue selecting the next round of training group and validation group data for training. In order to prevent overfitting, save computing resources, control training time, and balance accuracy and efficiency, a maximum number of training rounds can be set. If the set maximum number of training rounds is reached, the process ends, thereby obtaining a trained model for deployment.

[0197] After the process ends at S611 or S612, a trained howling detection model is obtained.

[0198] Through the above method, a trained howling suppression model and howling detection model are obtained. In the embodiment of the present application, the howling suppression model and the howling detection model are combined and applied to obtain a howling detection and suppression combined model. Howling suppression and howling detection are performed separately in the combined model, and the howling suppression output is weighted using the continuous prediction probability output of howling detection, so that the combined model output is smoother. The trained howling suppression model and howling detection model can be deployed together to various devices that need to perform howling suppression, such as Figure 1 The camera device 1 or terminal device 2 shown in the figure.

[0199] Figure 8 This is a flow chart of the howling suppression method provided in the embodiment of the present application. The method can be executed on a device (such as a camera device 1 or a terminal device 2) that is deployed with a merge model. Since the merge model used is compact in structure, small in size and has good real-time performance, it is particularly suitable for various embedded platforms. Figure 8 The howling suppression method according to the embodiment of the present application is described.

[0200] S801, obtaining audio data.

[0201] This step obtains the audio data that needs to be suppressed.

[0202] S802: Process the audio data through a howling suppression pre-processing module to obtain a first input audio.

[0203] S803: Input the first input audio into a howling suppression network to obtain a howling suppression mask.

[0204] S804: Process the audio data through a howling detection pre-processing module to obtain a second input audio.

[0205] S805, input the second input audio into a howling detection network to obtain a howling detection output.

[0206] S806, calculating the howling weight according to the howling detection output, the calculation formula is as follows:

[0207]

[0208] Among them, hs_mask is the howling suppression mask, hdout is the howling detection output, howl_weight is the howling weight, α and β are constants for controlling the howling detection sensitivity and howling suppression sensitivity. The howling weight is obtained by summing hdout and α, and the boundary of the howling weight is limited according to β.

[0209] By adding bias to the howling detection output through ɑ, ɑ may take values ​​of 0 to 1, 0 means directly using the howling detection output (low howling detection rate), 1 means the default howling signal always exists (high howling detection rate). By limiting the upper bound of the howling suppression weight through β, β may take values ​​of 0 to 1, 0 means no howling suppression, 1 means 100% howling suppression will be performed when howling is detected.

[0210] S807: Calculate a weighted howling suppression mask in the following manner:

[0211] hs_mask_weighted=howl_weight*hs_mask+1-howl_weight public

[0212] Formula 11: Wherein, hs_mask_weighted is the weighted howling suppression mask.

[0213] Depending on the values ​​of α and β, howl_weight may range from 0 to 1. According to formula 11, when howl_weight is 0, hs_maxk_weighted = 1, indicating that howling suppression is not performed; when howl_weight is 0.8, It means that 80% of the signal is suppressed, and the remaining 20% ​​keeps the original signal; when howl_weight is 1, Indicates that 100% of the signal is subjected to howling suppression. The howling weight is calculated according to the howling detection output, and the howling weight represents the continuous probability value of the howling detection output. It is used for howling suppression mask weighting to avoid the phenomenon that the howling suppression output audio is discontinuous due to the use of discrete detection labels.

[0214] S808: Process the audio data and the weighted howling suppression mask through a howling suppression post-processing module to obtain a final howling suppression output audio.

[0215] In the embodiment of the present application, a howling suppression network is used to perform howling suppression processing on audio data to obtain a howling suppression mask, a howling detection network is used to perform howling detection on the audio data to obtain a howling detection output, and the howling detection output of the howling detection model is used to perform weighted optimization on the howling suppression mask. The optimized howling suppression mask can accurately perform howling suppression on the audio data, thereby avoiding the problem of audio distortion in the howling suppression output.

[0216] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0217] like Fig. 9 As shown, the electronic device 900 may be as follows Figure 1 The camera device 1 or terminal device shown may also be other electronic devices with data processing capabilities. The electronic device 900 includes but is not limited to a processor 902 and a memory 904 .

[0218] The memory 904 is used to store a computer program 906. The memory 904 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory. The computer program 906 may include computer executable instructions.

[0219] The processor 902 is configured to execute a computer program 906 to implement the above-mentioned howling suppression method embodiment.

[0220] The processor 902 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0221] An embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned howling suppression method embodiment is implemented.

[0222] An embodiment of the present application provides a computer program, which can be executed by a processor to implement the above-mentioned howling suppression method embodiment.

[0223] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned howling suppression method embodiment is implemented.

[0224] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or other electronic device) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store computer program codes.

[0225] The algorithm or display provided here are not inherently related to any specific computer, virtual system or other equipment. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application.

[0226] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In the claims that list several devices, several units or modules in these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

[0227] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A howling suppression method, characterized in that: The method comprises: Get audio data; The audio data is pre-processed for howling suppression and then input into a howling suppression network to obtain a howling suppression mask; The audio data is processed before howling detection and then input into a howling detection network to obtain a howling detection output; weighting the howling suppression mask according to the howling detection output to obtain a weighted howling suppression mask; The audio data is post-processed with howling suppression according to the weighted howling suppression mask to obtain howling suppression output audio.

2. The method according to claim 1, characterized in that The step of weighting the howling suppression mask according to the howling detection output to obtain a weighted howling suppression mask comprises: Calculating a howling weight according to the howling detection output, a howling detection sensitivity control constant, and a howling suppression sensitivity control constant; The howling suppression mask is weighted according to the howling weight to obtain a weighted howling suppression mask.

3. The method according to claim 2, characterized in that The howling detection output is a probability value between 0 and 1, and the value ranges of the howling detection sensitivity control constant and the howling suppression sensitivity control constant are both 0 to 1; The calculating the howling weight according to the howling detection output, controlling the howling detection sensitivity constant and controlling the howling suppression sensitivity constant comprises: If the sum of the howling detection output and the controlled howling detection sensitivity constant is less than the controlled howling suppression sensitivity constant, determining the sum of the howling detection output and the controlled howling detection sensitivity constant as the howling weight; If the sum of the howling detection output and the control howling detection sensitivity constant is greater than or equal to the control howling suppression sensitivity constant, the control howling suppression sensitivity constant is determined as the howling weight.

4. The method according to claim 2 or 3, characterized in that: The step of weighting the howling suppression mask according to the howling weight to obtain a weighted howling suppression mask includes: The weighted howling suppression mask is calculated according to the following formula: hs_mask_weighted=howl_weight*hs_mask+1-howl_weight; Among them, hs_mask_weighted is the weighted howling suppression mask, howl_weight is the howling weight, and hs_mask is the howling suppression mask.

5. The method according to claim 1, characterized in that The howling detection network includes a feature extraction module, a recurrent neural network layer and a detection output module; The step of performing pre-howling detection processing on the audio data and then inputting the audio data into a howling detection network to obtain a howling detection output includes: After converting the audio data from the time domain to the frequency domain frame by frame, the amplitude of each frame is obtained, the logarithmic Mel spectrum of each frame is calculated according to the amplitude of each frame, and the logarithmic Mel spectrum of each frame is spliced ​​to obtain a splicing feature, wherein the dimensions of the splicing feature include data batch, feature channel, frame length and number of frames, and the feature channel includes an amplitude channel; The feature extraction module uses two-dimensional convolution to perform sliding window operation on the splicing feature in two dimensions of frame length and frame number to obtain a third feature; The recurrent neural network layer extracts features from the third feature according to temporal context information of the third feature to obtain a fourth feature; The detection output module uses a fully connected layer to reduce the dimension of the fourth feature in the channel dimension, and obtains the howling prediction probability value through an activation layer.

6. The method according to claim 1, characterized in that The howling suppression pre-processing is performed by a howling suppression pre-processing module, and the howling suppression post-processing is performed by a howling suppression post-processing module. The howling suppression pre-processing module, the howling suppression network and the howling suppression post-processing module constitute a howling suppression model. The processing process of the sample data of the howling suppression model includes: Get clean audio and impulse response; Segment the clean audio to obtain segment audio signals; Inputting the segment audio signal into the howling suppression model to obtain a howling suppression output of the segment audio signal; Randomly generate gain parameters and delay parameters; Inputting the howling suppression output, the impulse response and the gain parameter into a howling data generation module to obtain a segment howling signal; A delay is introduced for the segment howling signal according to the delay parameter, and the delayed segment howling signal is added to the segment audio signal in the clean audio at the same position as the delayed segment howling signal.

7. The method according to claim 1, characterized in that The howling detection pre-processing is performed by a howling detection pre-processing module. The howling detection pre-processing module and the howling detection network constitute a howling detection model. The processing process of the sample data of the howling detection model includes: Get clean audio and impulse response; Generating a random parameter set for each sub-block of the clean audio in blocks, wherein the random parameter set includes a howling introduction random parameter, a gain parameter, and a delay parameter; Initialize howling audio; Segment the clean audio to obtain segment audio signals; If it is determined that a howling suppression model is used to process the segment audio signal according to the howling introduction random parameter of the sub-block to which the segment audio signal belongs, the segment audio signal is input into the howling suppression model to obtain a howling suppression output of the segment audio signal, and the howling suppression output is used to replace the segment audio signal; Inputting the replaced segment audio signal, the impulse response and the gain parameter of the sub-block to which the segment audio signal belongs into a howling data generating module to obtain a segment howling signal; A delay is introduced for the segment howling signal according to the delay parameter of the sub-block to which the segment audio signal belongs, and the delayed segment howling signal is added to the segment audio signal in the howling audio at the same position as the delayed segment howling signal.

8. The method according to claim 7, characterized in that The processing process of the sample data of the howling detection model also includes: The delayed segment howling signal is added to a segment audio signal in the clean audio at the same position as the delayed segment howling signal.

9. The method according to claim 7, characterized in that: The processing process of the sample data of the howling detection model also includes: Divide the howling audio into frames to obtain a frame howling signal; Determining the frame energy of the frame howling signal; Counting the number of howling sampling points of the frame howling signal according to the frame energy; Marking a frame label of the frame howling signal according to a sampling point threshold and the number of howling sampling points; The frame labels of the howling signals of each frame in the howling audio are concatenated to obtain a howling label of the howling audio.

10. The method according to any one of claims 6 to 9, characterized in that: The processing flow of the howling data generating module includes: Calculate the convolution of the input audio signal and the impulse response to obtain the convolution output; The convolution output is normalized and multiplied by a gain parameter to obtain a processed convolution output; The processed convolution output is subjected to signal boundary value limitation to obtain a segment howling signal.

11. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the howling suppression method according to any one of claims 1 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the howling suppression method according to any one of claims 1 to 10 is implemented.

Images