Loudspeaker speech enhancement method and device, electronic equipment and storage medium

Abstract

The invention provides a loudspeaker speech enhancement method and device, electronic equipment and a storage medium, and belongs to the technical field of speech signal processing, and the method comprises the steps: obtaining an initial spectrum signal and an initial power spectrum signal corresponding to an initial audio signal; inputting the initial power spectrum signal into a feedback elimination network model to obtain a target frequency domain signal; and obtaining a target audio signal based on the target frequency domain signal. The method comprises the following steps: performing mask adjustment on the frequency domain amplitude of an initial frequency spectrum signal by using a real number signal mask to generate a frequency domain estimation signal, and performing mask adjustment on the frequency domain amplitude and phase of the initial frequency spectrum signal by using a complex number signal mask to obtain a frequency domain enhancement signal. And finally, a target frequency domain signal is determined based on the frequency domain estimation signal and the frequency domain enhancement signal, so that the output target frequency domain signal has higher fidelity in the aspects of amplitude and phase, environmental noise and loudspeaker feedback signals can be removed to the greatest extent, and the squeal phenomenon is effectively inhibited.

Description

Technical Field

[0001] The present invention relates to the field of speech signal processing technology, and in particular to a speaker speech enhancement method, device, electronic device and storage medium. Background Art

[0002] During the sound amplification process, speakers are often affected by ambient noise, room reverberation, and speaker feedback. This may cause the played voice to be distorted, sound unnatural, or even produce howling, which seriously affects voice quality.

[0003] Currently, directional microphones are commonly used to pick up sound from a specific direction, thereby suppressing noise and interference from other directions. However, directional microphones are essentially a passive defense measure, and because they lack effective intervention for sudden feedback signals during the sound amplification process, they are unable to address the howling phenomenon that has already occurred. Summary of the Invention

[0004] The present invention provides a speaker voice enhancement method, device, electronic device and storage medium to solve the defects of the prior art such as poor voice quality, large sound amplification delay and impact on user listening experience.

[0005] The present invention provides a method for enhancing speaker speech, comprising the following steps: Obtaining an initial frequency spectrum signal corresponding to the initial audio signal, and determining an initial power spectrum signal corresponding to the initial frequency spectrum signal; Inputting the initial power spectrum signal into a feedback cancellation network model to obtain a target frequency domain signal output by the feedback cancellation network model, wherein the feedback cancellation network model includes a first sub-model and a second sub-model; Obtaining a target audio signal based on the target frequency domain signal; The first sub-model outputs a real signal mask according to the input initial power spectrum signal; the real signal mask is used to perform mask adjustment on the frequency domain amplitude of the initial spectrum signal to obtain a frequency domain estimation signal; The second sub-model receives a three-dimensional spectrum signal obtained by splicing the frequency domain estimation signal and the initial spectrum signal, and outputs a complex signal mask; the complex signal mask is used to perform mask adjustment on the frequency domain amplitude and phase of the initial spectrum signal to obtain a frequency domain enhanced signal; The target frequency domain signal is determined based on the frequency domain estimation signal and the frequency domain enhanced signal.

[0006] According to a speaker speech enhancement method provided by the present invention, the first sub-model includes a first input module, a first GRU module, a second GRU module, a splicing module, a third GRU module and a first output module; The first input module receives the initial power spectrum signal and inputs the initial power spectrum signal into the first GRU module and the second GRU module in parallel; The first GRU module performs time series modeling on the initial power spectrum signal to obtain a first time series feature; the second GRU module performs time series modeling on the initial power spectrum signal to obtain a second time series feature; The splicing module splices the first time series feature and the second time series feature along the feature dimension to obtain a fused time series feature; The third GRU module receives the fused time series features to perform time series dependency modeling on the fused time series features to obtain final time series features; The first output module is used to map the final timing feature into the real signal mask.

[0007] According to a speaker speech enhancement method provided by the present invention, the second sub-model includes a second input module, an encoder module, a bottleneck processing module, a decoder module and a second output module; The second input module inputs the received three-dimensional spectrum signal into the encoder module; The encoder module includes a plurality of cascaded upsampling convolution layers, which are used to perform convolution and upsampling on the three-dimensional spectrum signal layer by layer and output intermediate coding features at each layer; The bottleneck processing module performs temporal context modeling on the intermediate coding features output by the last upsampling convolutional layer to perform filtering and weighting processing of adjacent time frames and output a bottleneck feature after temporal enhancement; The decoder module includes a plurality of cascaded downsampling convolution layers, which are used to downsample and convolve the bottleneck features layer by layer and output a frequency domain decoded signal; The second output module performs convolution mapping on the frequency domain decoded signal and outputs the complex signal mask.

[0008] According to a speaker speech enhancement method provided by the present invention, the bottleneck processing module includes a gated recurrent unit, a bidirectional gated recurrent unit and a linear mapping layer; The gated recurrent unit receives the intermediate coding feature output by the last upsampling convolutional layer of the encoder module, performs time series modeling on the intermediate coding feature along the time frame dimension and outputs a first time series feature; The bidirectional gated recurrent unit receives the first time series feature, performs bidirectional time series modeling to perform filtering and weighting processing of adjacent time frames, and outputs a second time series feature; The linear mapping layer performs mapping transformation on the second time series feature and outputs the bottleneck feature after time series enhancement.

[0009] According to a speaker speech enhancement method provided by the present invention, the downsampling convolution layer in the decoder module corresponds one-to-one to the upsampling convolution layer in the encoder module; In the process of downsampling and convolving the bottleneck features layer by layer in the decoder module and outputting the frequency domain decoded signal, the intermediate coding features with the same resolution output by the corresponding upsampling convolution layer are fused through jump connections in each downsampling convolution layer.

[0010] According to a speaker speech enhancement method provided by the present invention, the feedback elimination network model further includes a weighted fusion module; The weighted fusion module performs weighted fusion on the frequency domain estimation signal output by the first sub-model and the frequency domain enhanced signal output by the second sub-model to generate the target frequency domain signal.

[0011] According to a speaker speech enhancement method provided by the present invention, the initial audio signal is obtained based on the following method: Get the microphone receiving signal; The microphone received signal is sampled according to a preset synthesis window length and a preset frame shift to obtain the initial audio signal obtained by each sampling.

[0012] According to a speaker speech enhancement method provided by the present invention, the microphone received signal includes at least a target speech signal and a speaker feedback signal; After acquiring the microphone reception signal, eliminating the speaker feedback signal based on an adaptive feedback control algorithm specifically includes: Initializing an adaptive filter, wherein an error signal of the adaptive filter is set to a difference between the microphone reception signal and an estimated signal that predicts the loudspeaker feedback signal; updating the coefficients of the adaptive filter based on a preset adaptive algorithm to determine a minimized error signal; The minimized error signal is used as a new microphone reception signal after eliminating the loudspeaker feedback signal.

[0013] According to a method for enhancing speaker speech provided by the present invention, before eliminating the speaker feedback signal based on the adaptive feedback control algorithm, the method further includes: The microphone receiving signal is input into a hardware equalizer, so as to use the hardware equalizer to perform gain adjustment on an abnormal frequency band in the microphone receiving signal.

[0014] According to a method for enhancing speaker speech provided by the present invention, after obtaining a target audio signal based on the target frequency domain signal, the method further includes: performing frequency shift and phase modulation processing and / or half-wave rectification processing on the target audio signal; The frequency shift and phase modulation processing includes increasing or decreasing the spectrum of the target audio signal as a whole by a preset Hz, and periodically changing the phase of the target audio signal; The half-wave rectification process includes converting all sample values ​​smaller than zero in the time domain waveform of the target audio signal into sample values ​​greater than zero by taking absolute values.

[0015] According to a method for enhancing speaker speech provided by the present invention, after performing frequency shift and phase modulation processing and / or half-wave rectification processing on the target audio signal, the method further includes: The amplitude of the target audio signal is dynamically adjusted to control the dynamic range of the loudspeaker output signal.

[0016] The present invention also provides a speaker voice enhancement device, comprising the following modules: a signal acquisition unit, configured to acquire an initial spectrum signal corresponding to the initial audio signal, and determine an initial power spectrum signal corresponding to the initial spectrum signal; a signal processing unit, configured to input the initial power spectrum signal into a feedback cancellation network model to obtain a target frequency domain signal output by the feedback cancellation network model, wherein the feedback cancellation network model includes a first sub-model and a second sub-model; A signal generating unit is configured to obtain a target audio signal based on the target frequency domain signal.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-described speaker speech enhancement methods is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned speaker speech enhancement methods when executed by a processor.

[0019] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned speaker speech enhancement methods.

[0020] The speaker speech enhancement method, device, electronic device and storage medium provided by the present invention first use a real signal mask to mask and adjust the frequency domain amplitude of the initial spectrum signal to generate a frequency domain estimation signal, and then use a complex signal mask to mask and adjust the frequency domain amplitude and phase of the initial spectrum signal to obtain a frequency domain enhancement signal. Finally, the target frequency domain signal is determined based on the frequency domain estimation signal and the frequency domain enhancement signal, so that the output target frequency domain signal has higher fidelity in both amplitude and phase, can remove ambient noise and speaker feedback signals to the greatest extent, and effectively suppress howling. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a schematic diagram of the loudspeaker and microphone provided by the present invention receiving sound signals.

[0023] Figure 2 It is a flow chart of the loudspeaker speech enhancement method provided by the present invention.

[0024] Figure 3 It is a schematic diagram of the model architecture of the first sub-model provided by the present invention.

[0025] Figure 4 It is a schematic diagram of the model architecture of the second sub-model provided by the present invention.

[0026] Figure 5 Schematic diagram of the architecture of the upsampling convolutional layer provided by the present invention.

[0027] Figure 6 Schematic diagram of the architecture of the downsampling convolutional layer provided by the present invention.

[0028] Figure 7 It is a structural schematic diagram of the loudspeaker speech enhancement device provided by the present invention.

[0029] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0031] It should be noted that, in the description of the present invention, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method, article or device comprising the elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0032] The terms "first," "second," and the like in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that embodiments of the present invention can be implemented in orders other than those illustrated or described herein. Furthermore, the objects distinguished by "first," "second," and the like generally refer to a class of objects, and do not limit the number of objects. For example, the first object may be one or more.

[0033] The following combination Figures 1-8 The present invention describes a speaker speech enhancement method, device, electronic device, and storage medium.

[0034] Figure 1 Schematic diagram of the speaker and microphone provided by the present invention receiving sound signals, such as Figure 1 As shown, the microphone receives the signal in the sound amplification scenario It can be divided into three components: 1) Target speech : Direct speech from the speaker; 2) Feedback signal : The speaker plays the direct signal; 3) Background noise With room reverb : The noise in the environment and the reflection and delay of the speaker's voice and speaker signal in the room cause the sound to overlap and distort.

[0035] like Figure 1 As shown, the speaker plays the reference signal , then the feedback signal is played back to the microphone through the speaker As shown in formula (1): (1) in, Represents the feedback signal, represents the nonlinear transformation of the loudspeaker, Indicates that the speaker plays the reference signal, represents the propagation path from the speaker to the microphone, Represents linear convolution.

[0036] Feedback signal The signal is received by the microphone After delay and amplification, the feedback signal will be picked up by the microphone continuously, at any time The corresponding microphone receives the signal As shown in formula (2): (2) in, Indicates that the microphone receives the signal. represents the target voice, represents the background noise, represents room reverberation, represents the nonlinear transformation of the loudspeaker, represents the transmission delay from microphone to speaker, Indicates the total time for the feedback signal to be amplified. Indicates the speaker amplifier gain value, Represents the propagation path from the speaker to the microphone.

[0037] From formula (2), we can see that the feedback signal The repeated pickup will form positive feedback, resulting in cyclic amplification of signals at certain frequencies, which will cause howling.

[0038] Therefore, the present invention provides a method for enhancing speaker speech, the main purpose of which is to: On the one hand, it suppresses feedback signals and reverberation components in the microphone pickup to improve the clarity of the target speech; on the other hand, it reduces link latency to ensure that the time difference between the speaker's direct voice and the speaker's output signal is minimized, avoiding echo problems during the amplification process.

[0039] Figure 2 FIG. 1 is a flow chart of the speaker speech enhancement method provided by the present invention, as shown in FIG. Figure 2As shown, the speaker voice enhancement method provided by the present invention can be implemented by a speaker, a loudspeaker, a voice interaction device, a smart speaker, an electronic whiteboard with audio enhancement capabilities, an in-vehicle voice system, or other electronic devices with voice playback and processing capabilities. Unless otherwise specified, the subsequent embodiments will use a speaker as an example.

[0040] As an optional embodiment, the implementation steps of the speaker speech enhancement method mainly include but are not limited to: First, an initial frequency spectrum signal corresponding to the initial audio signal is obtained, and an initial power spectrum signal corresponding to the initial frequency spectrum signal is determined.

[0041] Specifically, the initial audio signal may be a time-domain speech signal collected by a microphone, which inevitably contains target speech, background noise, room reverberation, and feedback signals from a loudspeaker.

[0042] It should be noted that a time-domain speech signal refers to the waveform curve of sound changes over time, recording how the sound fluctuates from the first second to the last second. Considering that time-domain speech signals cannot be directly analyzed in the frequency domain or subjected to noise separation, the present invention converts them into frequency-domain signals to facilitate subsequent processing, such as separating the target speech from background noise and suppressing feedback signals.

[0043] Specifically, the initial spectrum signal refers to a frequency domain signal obtained by processing the initial audio signal through a short-time Fourier transform (STFT). The initial spectrum signal can be obtained by the following steps: (1) Frame the continuous initial audio signal, that is, divide it into multiple frames of audio signal according to the preset frame length and frame shift.

[0044] (2) Add a window to each frame of audio signal (for example, using a Hamming window or a Hanning window) to reduce spectral leakage.

[0045] (3) Perform Fast Fourier Transform (FFT) on each frame of the windowed audio signal to obtain the frequency domain representation corresponding to each frame.

[0046] (4) Finally, the frequency domains of all frames are concatenated to form the initial spectrum signal.

[0047] The initial spectrum signal obtained after the above processing is a complex signal, which completely retains the amplitude information and phase information of the original audio signal. For example, the initial spectrum signal can be used To indicate that Represents two channels, real and imaginary, represents the index of the timeframe, Indicates the index of the frequency point.

[0048] By reasonably setting the window length, frame shift, sampling rate and processing delay, the present invention can fully ensure that there is almost no obvious time difference between the playback and reception of the voice signal while meeting the requirements of real-time audio processing, thus avoiding the occurrence of echo or delay.

[0049] For example, in a preferred embodiment, the sampling rate can be set to 16000 Hz, the synthesis window length to 256 sampling points, and the frame shift to 80 sampling points. Based on these parameters, the processing delay is (window length + frame shift) / sampling rate = (256 + 80) / 16000 = 0.021 seconds, or 21 milliseconds. Such a low delay makes the time difference between the speaker's direct sound and the sound played through the speaker almost imperceptible to the human ear, effectively avoiding the common sense of echo and delay in sound amplification, significantly improving the user experience.

[0050] Specifically, determining the initial power spectrum signal corresponding to the initial spectrum signal refers to performing calculations based on the acquired initial spectrum signal to extract its energy distribution characteristics at different time frames and frequency points as input to the subsequent first sub-model.

[0051] Specifically, the obtained initial power spectrum signal is a real signal, which is calculated based on the amplitude information of the initial spectrum signal. It reflects the energy intensity of the initial audio signal at each time frame and frequency point, but does not contain phase information. Its calculation method can be shown as formula (3): (3) in, represents the initial power spectrum signal, and Represents the initial spectrum signal in the timeframe, There are two channels, real and imaginary, at each frequency point. Indicates in timeframe, The total power of the real and imaginary parts of the two channels at the frequency point, It means converting the calculated signal power into decibels (dB) in logarithmic form to conform to the human ear's perception of loudness changes, and also helps to stabilize the training and learning of subsequent neural network models.

[0052] Then, the initial power spectrum signal is input into the feedback cancellation network model to obtain the target frequency domain signal output by the feedback cancellation network model. The feedback cancellation network model mainly includes a first sub-model and a second sub-model.

[0053] The feedback cancellation network model is the core processing unit of the present invention and can be a deep neural network model trained on a large amount of data. The primary goal of the feedback cancellation network model is to intelligently estimate and suppress undesirable signal components (such as ambient noise, room reverberation, and speaker feedback) from the initial audio input signal, which includes the target speech, background noise, room reverberation, and speaker feedback, while preserving and enhancing the desired target speech, ultimately generating the target frequency domain signal.

[0054] The first sub-model outputs a real signal mask according to the input initial power spectrum signal.

[0055] Specifically, the first sub-model is composed of a specific neural network layer. Its design goal is to learn and extract the feature information related to the target speech by performing complex nonlinear transformation and time series modeling on the input initial power spectrum signal, and to distinguish the feature information of interference components such as noise and reverberation.

[0056] The first sub-model ultimately outputs a real-number signal mask, which is used to mask the frequency-domain amplitude of the initial spectrum signal to obtain a frequency-domain estimated signal. This real-number signal mask is a matrix or tensor with the same dimensions (i.e., the same number of time frames and frequency points) as the initial power spectrum signal, with each element being a real-valued value. For example, the range of these real-number values ​​is constrained to be between 0 and 1. In this case, a value close to 1 in the mask indicates that the model determines that the corresponding time-frequency unit primarily contains the target speech energy and should be retained or enhanced; a value close to 0 indicates that the time-frequency unit is determined to primarily contain interfering energy such as noise or reverberation and should be suppressed. Therefore, this real-number signal mask can be understood as an energy-based time-frequency gain filter.

[0057] After obtaining the real signal mask, it can be used to perform mask adjustment on the frequency domain amplitude of the initial spectrum signal. Specifically, mask adjustment involves element-by-element multiplication of the real signal mask with the amplitude spectrum of the initial spectrum signal. It should be noted that the initial spectrum signal is a complex signal, and the real signal mask only affects the amplitude portion of the initial spectrum signal, without changing its original phase information.

[0058] Specifically, the input of the first sub-model is the initial power spectrum signal, and the output is the real signal mask. The specific calculation method of the frequency domain estimation signal obtained by masking the frequency domain amplitude of the initial spectrum signal using the real signal mask is shown in formula (4): (4) in, represents the frequency domain estimated signal, represents the imaginary channel of the initial spectrum signal, Represents a real signal mask.

[0059] Compared with the initial spectrum signal, the phase information of the frequency domain estimated signal obtained after the above mask adjustment remains unchanged, but its amplitude spectrum has been modified by the real signal mask. The frequency domain amplitude dominated by the target speech component is retained, while the frequency domain amplitude dominated by the noise and reverberation components is effectively attenuated.

[0060] The second sub-model receives a three-dimensional spectrum signal obtained by splicing the frequency domain estimation signal and the initial spectrum signal, and outputs a complex signal mask.

[0061] Considering that the original initial spectrum signal contains complete, unprocessed mixed speech information, including the target speech and all types of interference, and the frequency domain estimation signal is the result of preliminary noise reduction and dereverberation processing by the first sub-model, it can be regarded as a preliminary estimation of the pure target speech. Therefore, the present invention obtains a three-dimensional spectrum signal by merging the frequency domain estimation signal and the initial spectrum signal. The obtained three-dimensional spectrum signal can provide the second sub-model with the above-mentioned key information carried by the initial spectrum signal and the frequency domain estimation signal respectively.

[0062] As an optional embodiment, the above-mentioned process of merging the frequency domain estimation signal and the initial spectrum signal to obtain a three-dimensional spectrum signal is specifically to splice the two channels of the real part and the imaginary part of the initial spectrum signal with the channel of the amplitude spectrum of the frequency domain estimation signal in the channel dimension, so as to obtain an input tensor with three channels, that is, a three-dimensional spectrum signal.

[0063] When making decisions about the three-dimensional spectrum signal, the second sub-model not only analyzes the original, interfering initial spectrum signal but also references a frequency domain estimation signal that has been preliminarily cleaned. This allows it to more accurately learn how to further suppress residual interfering components, particularly feedback and reverberation signals that are similar in energy to the target speech but differ in phase, and outputs a complex signal mask.

[0064] It's important to emphasize that the complex signal mask output by the second sub-model is fundamentally different from the real signal mask output by the first sub-model. It is a complex tensor with the same dimensions as the initial spectral signal, and each element contains both real and imaginary parts, enabling simultaneous adjustment of the frequency-domain amplitude and phase of a complex signal.

[0065] After obtaining the complex signal mask, the next step is to perform mask adjustment on the frequency domain amplitude and phase of the initial spectrum signal. The complex signal mask is used to mask the frequency domain amplitude and phase of the initial spectrum signal to obtain a frequency domain enhanced signal. Mask adjustment is achieved through complex multiplication, which involves element-by-element multiplication of the complex signal mask with the original initial spectrum signal.

[0066] It should be noted that the complex mask here is applied to the initial spectrum signal rather than the frequency domain estimation signal output by the first stage. This can avoid the accumulation of errors and allow the second stage to make corrections directly based on the original signal, thereby obtaining more accurate enhancement results.

[0067] As an optional embodiment, a complex signal mask is applied to the initial spectrum signal to obtain a frequency domain enhanced signal. The specific calculation method is shown in formula (5) and formula (6): (5) (6) in, represents the three-dimensional spectrum signal, represents the imaginary channel of the initial spectrum signal, represents the frequency domain estimated signal, represents the frequency domain enhanced signal, Represents a complex signal mask.

[0068] Compared with the frequency domain estimation signal output by the first stage, the final frequency domain enhanced signal is not only optimized in frequency domain amplitude (that is, the energy of interference such as noise, reverberation, and feedback is suppressed), but also its phase is corrected, making it closer to the phase of the pure target speech.

[0069] The following is a brief introduction to the specific training process of the feedback elimination network model.

[0070] The training samples for the feedback cancellation network model can be multiple audio signal samples collected by a microphone. These audio signal samples also include speech signals, ambient noise, room reverberation, and speaker feedback signals. A short-time Fourier transform (SFT) of the audio signal samples yields corresponding spectral signal samples. The power spectrum signal samples are then calculated based on the amplitude information of the spectral signal samples and used as input sample data for the feedback cancellation network model.

[0071] Furthermore, the training label of the feedback cancellation network model can be annotated as a frequency domain signal label corresponding to the audio signal sample. It should be noted that the frequency domain signal label is usually a theoretically pure speech spectrum that has been preprocessed or recorded in a controlled environment. It represents the frequency domain representation of the pure target speech after completely removing the environmental noise, room reverberation and feedback signal. The frequency domain signal label serves as a supervisory signal to guide the feedback cancellation network model to learn during the training process. By adjusting the relevant model parameters of the first sub-model and the second sub-model, it enables them to learn effective frequency domain amplitude and phase adjustment strategies respectively, thereby achieving accurate suppression of noise and feedback signals.

[0072] As an optional embodiment, the training process of the feedback cancellation network model mainly includes but is not limited to the following steps: (1) Training data preparation: Obtain audio signal samples collected by the microphone and perform a short-time Fourier transform on them to obtain complex spectrum signal samples. Then, calculate the power based on the real and imaginary parts of the spectrum signal samples to generate power spectrum signal samples, which are used as input to the network model. At the same time, prepare the frequency domain signal labels representing the pure target speech as training labels. Furthermore, the input data can be normalized to improve data consistency and the stability of model learning.

[0073] (2) Data augmentation: To improve the robustness and generalization ability of the model, data augmentation techniques can be used to expand the diversity of training samples. For example, audio signal samples can be randomly shifted in time, spectrum signal samples can be scaled in frequency, or background noise and simulated reverberation of different types and intensities can be superimposed on the signal to enhance the model's adaptability to interference factors in various practical application environments.

[0074] (3) Loss function design: Construct a loss function for supervised model training. This loss function is used to measure the difference between the model output and the training label (the ideal target frequency domain signal). For example, the traditional mean squared error (MSE) can be combined with one or more auditory perception-based evaluation indicators (such as loss related to speech quality perception assessment), thereby optimizing the overall auditory quality of the final output audio while optimizing the frequency domain restoration accuracy.

[0075] (4) Back propagation and parameter optimization: During the training iteration process, the gradient is calculated by the back propagation algorithm based on the error calculated by the loss function, and an efficient optimization algorithm (such as the Adam optimization algorithm) is used to update all learnable parameters in the feedback elimination network model.

[0076] In addition, the learning rate decay mechanism can be combined to use a larger learning rate in the early stage of training to accelerate convergence, and a smaller learning rate in the later stage of training to help the model converge to a better point. This helps to improve the convergence speed of the training process while reducing the risk of overfitting, thereby improving the stability and performance of the model in practical applications.

[0077] The following briefly describes the relevant implementation steps of determining the target frequency domain signal based on the frequency domain estimation signal and the frequency domain enhancement signal, and obtaining the target audio signal according to the target frequency domain signal.

[0078] First, the frequency domain estimation signal and the frequency domain enhancement signal are the products of two different processing strategies in two stages, each with its own advantages. The frequency domain estimation signal effectively suppresses most noise and reverberation while preserving the original phase through real-valued masking, but may not be able to process feedback signals that are highly correlated with the target speech. The frequency domain enhancement signal, on the other hand, uses complex-valued masking to jointly optimize amplitude and phase, enabling more precise processing of residual interference, but may also introduce unnecessary processing artifacts. Therefore, determining the target frequency domain signal involves fusing the frequency domain estimation signal and the frequency domain enhancement signal.

[0079] As an optional embodiment, the final target frequency domain signal can be determined by weighted fusion of the frequency domain estimation signal and the frequency domain enhancement signal. For example, the feedback cancellation network model can additionally output a time-varying weighting coefficient, which is used to dynamically adjust the proportion of the two signals in the final output. In certain time frames, when the output quality of the first sub-model is higher, it can be assigned a greater weight; conversely, the output of the second sub-model is assigned a greater weight. Through this adaptive fusion strategy, it can be ensured that the final target frequency domain signal achieves the optimal enhancement effect in all situations.

[0080] As an optional embodiment, the target frequency domain signal may be calculated as shown in formula (7): (7) in, is the target frequency domain signal after fusion, is the weighting coefficient, represents the frequency domain estimation signal of the first stage, Represents the frequency domain enhanced signal of the second stage.

[0081] Furthermore, after determining the target frequency domain signal, the target audio signal is derived from it. This involves converting the signal from the frequency domain back to the time domain, generating an enhanced time domain waveform signal that can be directly played by speakers. This can be achieved using an inverse short-time Fourier transform (ISTFT).

[0082] Specifically, the specific implementation steps of the inverse short-time Fourier transform can be: (1) Perform an inverse fast Fourier transform (IFFT) on each frame of the target frequency domain signal to convert it from the frequency domain representation back to the time domain waveform.

[0083] (2) All the short-time time domain waveforms obtained by inverse transformation are spliced ​​and synthesized by overlapping and adding methods to restore a continuous and complete time domain audio signal, which is the target audio signal.

[0084] The speaker speech enhancement method provided in this embodiment first uses a real signal mask to mask and adjust the frequency domain amplitude of the initial spectrum signal to generate a frequency domain estimation signal, and then uses a complex signal mask to mask and adjust the frequency domain amplitude and phase of the initial spectrum signal to obtain a frequency domain enhancement signal. Finally, the target frequency domain signal is determined based on the frequency domain estimation signal and the frequency domain enhancement signal, so that the output target frequency domain signal has higher fidelity in both amplitude and phase, can remove ambient noise and speaker feedback signals to the greatest extent, and effectively suppress howling.

[0085] Figure 3 is a schematic diagram of the model architecture of the first sub-model provided by the present invention, such as Figure 3 As shown, as an optional embodiment, the first sub-model includes a first input module, a first GRU module, a second GRU module, a splicing module, a third GRU module and a first output module.

[0086] In actual application, the first input module receives the initial power spectrum signal and inputs the initial power spectrum signal into the first GRU module and the second GRU module in parallel.

[0087] The first input module serves as the data entry for the first sub-model, and its function is to receive the initial power spectrum signal calculated in the above steps. Figure 3 As shown in Figure 1, the input initial power spectrum signal is distributed to the first GRU module and the second GRU module simultaneously and indiscriminately by the first input module. This parallel processing structure allows the model to use different parameter groups to analyze the same input data, thereby extracting more diverse features.

[0088] Below, we will briefly introduce the implementation process of two parallel GRU modules processing the input signals respectively.

[0089] The first GRU module performs time series modeling on the initial power spectrum signal to obtain a first time series feature; the second GRU module performs time series modeling on the initial power spectrum signal to obtain a second time series feature.

[0090] The Gated Recurrent Unit (GRU) is a special type of recurrent neural network that excels at capturing temporal dependencies in sequential data, such as time frames in audio signals. Through its internal update and reset gate mechanisms, the GRU learns which information to retain and which to forget across time.

[0091] Specifically, time series modeling refers to the GRU module processing the initial power spectrum signal along the time frame dimension and analyzing the correlation between the spectrum characteristics of the current time frame and the characteristics of the previous and next time frames.

[0092] For example, the first GRU module and the second GRU module can be set to have different internal parameters or structures, such as different numbers of hidden layer units, or one can be a unidirectional GRU and the other a bidirectional GRU. Through such a setting, they can respectively learn temporal context information at different scales or in different directions, thereby generating two sets of features with complementary content but different representations, namely the first temporal features and the second temporal features.

[0093] After obtaining two sets of parallel time series features, the splicing module splices the first time series features and the second time series features along the feature dimension to obtain a fused time series feature.

[0094] Specifically, the concatenation module performs a feature fusion operation along the feature dimension. Assuming the dimensions of both the first and second time series features are 128, the dimension of the concatenated fused time series features will be 256. Feature fusion integrates the diverse time series information learned from the two parallel branches, forming a more comprehensive and informative feature representation.

[0095] Then, the fused time series features obtained after the fusion of the above features are further deeply processed, that is, the fused time series features are received by the third GRU module to perform time series dependency modeling on the fused time series features to obtain the final time series features. This can be understood as a further integration and abstraction of the fused time series features that have been preliminarily refined, thereby capturing longer-term and more complex time series associations.

[0096] Finally, the final time series features are mapped to a real signal mask through the first output module. The first output module can be composed of one or more fully connected layers and an activation function (such as a Sigmoid function). It receives the final time series features output by the third GRU module and maps them to the same size as the initial power spectrum signal through a linear transformation. The Sigmoid activation function set in the first output module can constrain the output value between 0 and 1, so that its physical meaning is consistent with the gain mask. This results in a real signal mask that can be directly used to mask the frequency domain amplitude of the initial spectrum signal.

[0097] The speaker speech enhancement method provided by the present invention, through a structure including a parallel GRU module and a serial GRU module connected together, enables the first sub-model to effectively perform deep temporal context modeling on the initial power spectrum signal. This structural design not only extracts diverse temporal features through parallel branches, but also integrates these features through subsequent splicing and deep modeling, thereby more accurately estimating the energy distribution difference between the target speech and the interference, and ultimately generating a high-quality real signal mask, providing strong support for achieving efficient noise and reverberation suppression.

[0098] Figure 4 is a schematic diagram of the model architecture of the second sub-model provided by the present invention, such as Figure 4 As shown, the second sub-model mainly includes a second input module, an encoder module, a bottleneck processing module, a decoder module and a second output module.

[0099] Specifically, the second input module serves as the data entry of the second sub-model, which receives a three-dimensional spectrum signal spliced ​​by the initial spectrum signal and the frequency domain estimation signal along the channel dimension, and passes the three-dimensional spectrum signal to the subsequent encoder module for feature extraction.

[0100] Next, the encoder module analyzes the input 3D spectrum signal layer by layer. This encoder module functions as a feature extractor and consists of multiple cascaded upsampling convolutional layers. In each upsampling convolutional layer, convolution is used to extract local features. Pooling is used to reduce the resolution of the feature map in both the time and frequency dimensions, while increasing the number of feature channels.

[0101] For example, after the input 3D spectrum signal passes through the first upsampling convolutional layer, the number of channels increases from 3 to 8; after the second upsampling convolutional layer, the number of channels can increase to 16, and so on. This process can gradually capture signal features from concrete to abstract. The features output by each upsampling convolutional layer in the encoder module are intermediate encoded features, which are saved for use in subsequent jump connections in the decoder module.

[0102] The bottleneck processing module is used to perform temporal context modeling on the intermediate encoding features output by the last upsampling convolutional layer in the encoder module to perform filtering and weighted processing of adjacent time frames and output the bottleneck features after temporal enhancement.

[0103] Specifically, the bottleneck processing module is located at the connection between the encoder module and the decoder module, and processes the intermediate coding features with the lowest resolution but the richest semantic information. The core of the bottleneck processing module is the recurrent neural network unit, and its temporal context modeling is achieved by expanding the input feature map along the time frame dimension and inputting it into the GRU or bidirectional GRU. The weighted filtering processing of adjacent time frames is completed through the gating mechanism of the GRU or bidirectional GRU, which can learn the dependency relationship between the features of the current time frame and the features of the previous and next adjacent time frames, and perform weighted fusion of the dependency relationship between the features of the current time frame and the features of the previous and next adjacent time frames, and finally output the bottleneck features enhanced by the temporal information.

[0104] The decoder module consists of multiple cascaded downsampling convolutional layers, which reconstruct the bottleneck features output by the bottleneck processing module. In each downsampling convolutional layer, transposed convolution is used to increase the resolution of the feature map in both the time and frequency dimensions, while reducing the number of channels, thereby gradually restoring the feature map to the same resolution as the input signal. The decoder module uses the bottleneck features output by the bottleneck processing module as its initial input and decodes them layer by layer through multiple downsampling convolutional layers, ultimately outputting a frequency-domain decoded signal.

[0105] Finally, the second output module performs convolution mapping on the frequency domain decoded signal and outputs a complex signal mask.

[0106] Specifically, the second output module can be a 1×1 convolution layer, which receives the frequency domain decoded signal output by the decoder module, and maps the number of frequency domain decoded signal channels to 2 by performing the final convolution mapping, corresponding to the real part and imaginary part of the complex signal mask, thereby obtaining the final complex signal mask.

[0107] The speaker speech enhancement method provided by the present invention utilizes an encoder module to extract intermediate coding features from a three-dimensional spectrum signal. The bottleneck processing module then performs temporal dependency modeling on the intermediate coding features and outputs bottleneck features enhanced with temporal information. Finally, the decoder module reconstructs the bottleneck features output by the bottleneck processing module into a high-resolution mask, providing powerful model support for generating high-quality complex signal masks, thereby significantly improving the performance of speech enhancement.

[0108] Figure 5 Schematic diagram of the architecture of the upsampling convolution layer provided by the present invention, which is the basic unit constituting the encoder module. Figure 5 As shown in Figure 1, the upsampling convolution layer starts from the input layer, and its core part is mainly composed of two sequentially cascaded convolution-batch normalization layers and an activation function.

[0109] Specifically, the processing flow of the upsampling convolution layer includes but is not limited to the following steps: (1) The input layer receives the input feature map and passes it to the first cascaded convolution-batch normalization layer. The convolution layer performs a convolution operation on the input feature map through its internal convolution kernel to extract local features. Then, the batch normalization layer normalizes the feature map output by the convolution to adjust the data distribution, thereby improving the stability and convergence speed of model training.

[0110] (2) The feature map processed by the first cascade convolution-batch normalization layer is then sent to the second cascade convolution-batch normalization layer, whose structure also includes a convolution layer and a batch normalization layer, aiming to perform deeper feature extraction.

[0111] (3) After the feature map is processed by the second cascaded convolution-batch normalization layer, it is further transformed nonlinearly through the ReLU activation function, setting all negative elements to zero and retaining only positive elements, thereby enhancing the expressive power of the model and helping to alleviate the gradient vanishing problem.

[0112] Figure 6 This is a schematic diagram of the architecture of the downsampling convolutional layer provided by the present invention, such as Figure 6 As shown in the figure, the downsampling convolution layer mainly includes the input layer, convolution layer, batch normalization layer, transposed convolution layer, second batch normalization layer, ReLU activation function layer and output layer.

[0113] Specifically, the processing flow of the downsampling convolution layer includes but is not limited to the following steps: (1) The input layer receives the feature map from the previous layer and passes it to the convolution layer for preliminary feature extraction. The convolution layer performs a convolution operation on the input feature map through multiple convolution kernels to extract local feature information.

[0114] (2) The feature map output by the convolutional layer is passed to the batch normalization layer, which normalizes the feature map, adjusts the data distribution, and reduces the gradient fluctuation during training, thereby improving the stability and convergence speed of model training.

[0115] (3) The normalized feature map is input into the transposed convolution layer, which implements the spatial downsampling operation of the feature map, improves the time and frequency dimension resolution of the feature map, and provides conditions for subsequent fine feature recovery.

[0116] (4) The output of the transposed convolutional layer undergoes a second batch normalization process to further standardize the data distribution and ensure the effectiveness of subsequent nonlinear transformations.

[0117] (5) Subsequently, the feature map undergoes a nonlinear transformation through the ReLU activation function layer, setting all negative values ​​to zero and retaining only positive elements, thereby enhancing the nonlinear expression capability of the model and effectively alleviating the gradient vanishing problem.

[0118] (6) Finally, the feature map processed as above is output by the output layer and passed to the next processing unit of the decoder module as the processing result of the downsampling convolution layer.

[0119] In another embodiment provided by the present invention, the bottleneck processing module includes a gated recurrent unit, a bidirectional gated recurrent unit and a linear mapping layer; receives the intermediate coding features output by the last level upsampling convolutional layer of the encoder module, performs time series modeling on it along the time frame dimension and outputs the first time series features.

[0120] Specifically, the intermediate encoded features have the lowest resolution in the time and frequency dimensions, but the highest feature channel dimension, embodying the most abstract, high-level semantic information. The gated recurrent unit iteratively processes the feature map along the time frame sequence, leveraging its internal gating mechanism to learn and capture the forward temporal dependencies of the signal. After this unidirectional temporal processing, it outputs the first temporal feature.

[0121] Next, a deeper level of modeling is performed on the preliminary time series feature. A bidirectional gated recurrent unit receives the first time series feature, performs bidirectional time series modeling to perform filtering and weighting processing of adjacent time frames, and outputs a second time series feature.

[0122] Specifically, the bidirectional gated recurrent unit consists of a forward GRU and a backward GRU. After receiving the first time series features, the forward GRU processes from the beginning to the end of the sequence, while the backward GRU processes from the end to the beginning of the sequence. This bidirectional time series modeling enables the second sub-model to simultaneously utilize its past and future information when processing any time frame. Through this bidirectional mechanism, the second sub-model can weight and filter the features of each time frame based on the complete contextual information, thereby more effectively identifying and modeling signal components with specific time evolution patterns. After processing, the second time series features output by the bidirectional gated recurrent unit contain richer and more comprehensive time series contextual information.

[0123] Finally, the linear mapping layer performs mapping transformation on the second temporal features and outputs the bottleneck features after temporal enhancement.

[0124] Specifically, the linear mapping layer can be a fully connected layer that receives the second time series features output from the bidirectional gated recurrent unit. The linear mapping layer performs a linear transformation on the second time series features, and the parameters of the mapping transformation are learned by the second sub-model during the training process.

[0125] The speaker speech enhancement method provided by this invention utilizes a bottleneck processing module consisting of a serially cascaded gated recurrent unit, a bidirectional gated recurrent unit, and a linear mapping layer. This module performs a comprehensive temporal analysis of the intermediate encoded features extracted by the encoder, from shallow to deep, and from unidirectional to bidirectional. This structural design enhances the model's ability to capture and understand the temporal dynamics of the signal, particularly for processing feedback and reverberation signals that are strongly temporally correlated with the target speech. This provides a solid foundation for the decoder module to accurately reconstruct the signal and ultimately generate a high-quality complex signal mask.

[0126] In another embodiment provided by the present invention, the downsampling convolutional layers in the decoder module correspond one-to-one to the upsampling convolutional layers in the encoder module.

[0127] Specifically, this one-to-one correspondence is the basis for building a U-shaped network symmetric structure, that is, a layer at a certain resolution level in the encoder module must have a corresponding layer at the same resolution level in the decoder module.

[0128] In the process of downsampling and convolving the bottleneck features layer by layer in the decoder module and outputting the frequency domain decoding signal, the intermediate coding features with the same resolution output by the corresponding upsampling convolution layer are fused through jump connections in each downsampling convolution layer.

[0129] Specifically, the fusion process may include but is not limited to the following steps: (1) A downsampling convolutional layer in the decoder module first downsamples the feature map received from the previous layer to improve its resolution in the time and frequency dimensions.

[0130] (2) Through skip connections, the intermediate encoded features of the corresponding level in the encoder module with the same resolution are directly passed to the downsampling convolutional layer.

[0131] (3) The downsampled feature map and the intermediate encoding features passed through the jump connection are spliced ​​in the channel dimension to achieve feature fusion.

[0132] The speaker speech enhancement method proposed in this paper employs skip connections to fuse coded features of the same resolution. This allows the decoder module to directly utilize the precise position and detail information captured by the encoder module during upsampling when reconstructing the signal. This enables the resulting complex signal mask to more accurately restore the spectral structure of the target speech, effectively avoiding speech distortion or blurring caused by information loss. This significantly improves the quality of the frequency-domain enhanced signal and the clarity and fidelity of the final output target audio signal.

[0133] In another embodiment provided by the present invention, the feedback elimination network model also includes a weighted fusion module; the weighted fusion module performs weighted fusion on the frequency domain estimation signal output by the first sub-model and the frequency domain enhancement signal output by the second sub-model to generate a target frequency domain signal.

[0134] Considering the frequency domain estimation signal output by the first sub-model, interference is suppressed mainly by adjusting the frequency domain amplitude, which retains the original phase; while the frequency domain enhanced signal output by the second sub-model jointly adjusts the frequency domain amplitude and phase through complex masking, which can process the residual reverberation and feedback signals more finely, but may also introduce imperceptible processing distortion.

[0135] Therefore, the present invention provides the weighted fusion module to comprehensively utilize the respective processing advantages of the two sub-models.

[0136] Specifically, the weighted fusion process can be controlled by one or more learnable weighting coefficients. These weighting coefficients can be learned and output by the feedback cancellation network model itself, for example, as an additional output of the second sub-model. These weighting coefficients can be dynamic values ​​that change over time, used to dynamically adjust the weights of the two signals in the final output.

[0137] The present invention considers the frequency domain estimation signal output by the first sub-model and suppresses interference mainly by adjusting the frequency domain amplitude. This has the advantage of preserving the original phase. The frequency domain enhancement signal output by the second sub-model is jointly adjusted in the frequency domain amplitude and phase through complex masking, which can more finely process residual reverberation and feedback signals, but may sometimes introduce subtle processing distortion.

[0138] The speakerphone speech enhancement method proposed in this paper avoids the limitations of a single model by integrating the outputs of two stages using a weighted fusion module. This design enables the feedback cancellation network model to dynamically combine the processing results of the two sub-models based on the signal content, thereby generating a target frequency domain signal that achieves an optimal balance among speech clarity, noise suppression, feedback cancellation, and natural listening, further enhancing the stability and robustness of the entire speakerphone speech enhancement system.

[0139] In another embodiment provided by the present invention, the initial audio signal is obtained based on the following method: obtaining a microphone reception signal, sampling the microphone reception signal according to a preset synthesis window length and a preset frame shift to obtain the initial audio signal.

[0140] Specifically, the microphone received signal is the raw digital audio stream captured by the microphone hardware in the speaker system and converted from analog to digital. It should be noted that the microphone received signal is a continuous time-domain signal stream that reflects all sound components in the sound field surrounding the microphone in real time, including the target speech, background ambient noise, reverberation caused by room reflections, and feedback signals generated by the speaker after playback.

[0141] Next, in order to process the continuous time domain signal stream, the microphone received signal needs to be framed according to a preset synthesis window length and a preset frame shift.

[0142] Specifically, the framing process is to divide the continuous signal stream into multiple overlapping data blocks, namely audio frames.

[0143] The preset synthesis window length defines the length of each audio frame, in units of sampling points, which determines the frequency resolution during subsequent frequency domain analysis.

[0144] The preset frame shift defines the distance between the starting points of two adjacent audio frames, also in samples. This preset frame shift is smaller than the synthesis window length, allowing for overlap between adjacent frames. This overlap ensures that information at frame boundaries is not lost during framing, and contributes to smoother audio during signal reconstruction.

[0145] For example, in a scenario with a sampling rate of 16000 Hz, the preset synthesis window length can be set to 256 sampling points, and the preset frame shift can be set to 80 sampling points. The speaker speech enhancement method provided by the present invention successfully converts a continuous, infinitely long audio stream into a discrete, serialized initial audio signal sequence by adopting this method of segmented acquisition of the microphone received signal based on the synthesis window length and frame shift. This processing method is the basis for realizing real-time streaming speech enhancement, which enables the incoming audio data to be analyzed and enhanced instantly without waiting for the end of the entire speech segment, thereby ensuring the low-latency characteristics of the speaker speech enhancement method and providing a smooth, echo-free user experience for real-time sound reinforcement scenarios.

[0146] In another embodiment provided by the present invention, the microphone received signal includes at least a target speech signal and a speaker feedback signal; after obtaining the microphone received signal, the speaker feedback signal is eliminated based on an adaptive feedback control algorithm, specifically including: initializing an adaptive filter, and the error signal of the adaptive filter is set to the difference between the microphone received signal and an estimated signal that predicts the speaker feedback signal.

[0147] It should be noted that the present application uses an adaptive feedback control (AFC) algorithm to process a signal such as speaker feedback that is linearly correlated with a reference signal (ie, a signal played by the speaker).

[0148] Specifically, the execution process of the AFC algorithm includes but is not limited to the following steps: (1) Initialize the adaptive filter and set the error signal. The adaptive filter is a digital filter whose filter coefficients can be dynamically adjusted based on the statistical characteristics of the input signal. Its coefficients can be set to zero or a small random value during initialization. The AFC algorithm requires a reference signal, which is the signal being sent to the speaker for playback. By filtering this reference signal through the current adaptive filter, an estimated signal that predicts the speaker feedback signal can be obtained. The error signal is set to the difference between the real-time microphone received signal and the estimated signal.

[0149] (2) Update the filter coefficients based on a preset adaptive algorithm. The preset adaptive algorithm may be a Normalized Least Mean Squares (NLMS) algorithm. The NLMS algorithm uses the error signal calculated in the previous step to iteratively adjust the coefficients of the adaptive filter, with the goal of minimizing the energy of the subsequently calculated error signal. Over time, the coefficients of the adaptive filter will continue to converge, making the simulated acoustic path closer and closer to the actual physical path, thereby determining a minimized error signal.

[0150] (3) The minimized error signal is used as the new microphone reception signal. When the adaptive filter converges or reaches a stable state, the error signal it outputs is the signal after feedback cancellation processing. The speaker feedback signal component in this signal has been cancelled to the greatest extent. At this point, the signal that has undergone preliminary cleaning, that is, the new microphone reception signal, will replace the original microphone reception signal as the input for subsequent processing (such as frame sampling).

[0151] The speaker speech enhancement method proposed in this paper pre-processes microphone signals using an adaptive feedback control algorithm. This method uses efficient traditional signal processing methods to largely eliminate linearly dependent speaker feedback signals before they enter the complex deep learning network. This design reduces the processing burden of the subsequent feedback cancellation network model, allowing it to focus on processing nonlinear, more complex reverberation and background noise, thereby improving the stability of the entire speech enhancement system and the ultimate enhancement effect.

[0152] Taking into account different amplification equipment (including microphones, speakers) and usage environments, it will cause them to have excessively high gain peaks in specific frequency bands. This abnormal frequency band can easily be rapidly amplified in the feedback loop, resulting in howling.

[0153] Therefore, in another embodiment provided by the present invention, before eliminating the speaker feedback signal based on the adaptive feedback control algorithm, it also includes: inputting the microphone receiving signal into a hardware equalizer (EQ) to use the hardware equalizer to adjust the gain of the abnormal frequency band in the microphone receiving signal.

[0154] Specifically, the hardware equalizer is a hardware circuit or device that can independently adjust the gains of different frequency components in an audio signal and is connected in series between a microphone and a subsequent adaptive feedback control processing unit.

[0155] This gain adjustment process can be a pre-configured calibration operation. For example, before a device leaves the factory or is used for the first time in a specific environment, the device's frequency response can be measured using a test signal to identify the abnormal frequency band. The hardware equalizer can then be configured to apply gain reduction specifically to the identified abnormal frequency band. This way, when the original microphone signal passes through the hardware equalizer, the energy in these high-risk frequency bands is pre-attenuated.

[0156] The speaker voice enhancement method proposed in this invention proactively suppresses the inherent risk of howling from the very beginning, right before the signal enters the processing system, by adding a hardware equalizer calibration step before the adaptive feedback control algorithm. This design not only significantly enhances the stability of the entire sound amplification system but also provides the subsequent adaptive feedback control algorithm with a more stable input signal with improved frequency response characteristics, helping the AFC algorithm converge faster and operate more accurately.

[0157] Considering that despite being processed by the feedback cancellation network model, the output target audio signal is still highly similar to the original target speech in content, and the strong correlation between the return signal after being played by the speaker and the new target speech signal still exists, there is a risk of establishing an acoustic feedback loop.

[0158] Therefore, in another embodiment provided by the present invention, after obtaining the target audio signal based on the target frequency domain signal, it also includes: performing frequency shifting and phase modulation processing and / or half-wave rectification processing on the target audio signal. This step is intended to actively destroy the strong correlation between the feedback signal and the target speech signal by changing the physical properties of the signal.

[0159] Specifically, this step may include but is not limited to the following two processing methods: (1) Frequency shift and phase modulation processing: This processing fine-tunes the signal in the frequency domain and phase. It includes increasing or decreasing the spectrum of the target audio signal by a preset Hertz value (e.g., 5 Hertz) to disrupt the precise frequency alignment of the feedback signal and the original signal; and periodically changing the phase of the target audio signal to disrupt the phase consistency between the two. This processing has minimal impact on human hearing, but it can effectively weaken the conditions for establishing the feedback loop.

[0160] (2) Half-wave rectification: This process performs nonlinear transformation on the signal waveform in the time domain. It involves converting all sample values ​​less than zero in the time domain waveform of the target audio signal into sample values ​​greater than zero by taking their absolute values. This nonlinear operation significantly changes the waveform structure of the signal, effectively destroying the linear correlation between the feedback signal and the original signal.

[0161] It should be noted that the frequency shift and phase modulation processing and the half-wave rectification processing can be performed separately or in combination to achieve a balance according to actual anti-howling needs and sound quality requirements.

[0162] The speaker speech enhancement method provided by this invention performs decorrelation processing on the target audio signal to be played, resulting in the sound emitted from the speaker having physical characteristics that are significantly different from the original target speech. This design fundamentally destroys the strong correlation between the feedback signal and the target speech signal, making it difficult to establish positive feedback, thereby significantly improving the stability of the sound amplification equipment.

[0163] Considering that after multiple stages of signal processing, the amplitude of the target audio signal may become unpredictable or its dynamic range may be too wide, there is a risk of causing hardware clipping distortion or unstable output volume.

[0164] Therefore, in another embodiment provided by the present invention, after performing frequency shift and phase modulation processing and / or half-wave rectification processing on the target audio signal, it also includes: dynamically adjusting the amplitude of the target audio signal to control the dynamic range of the speaker output signal.

[0165] Specifically, controlling the dynamic range of the speaker output signal may include but is not limited to the following operations: (1) Compression: When the amplitude of the target audio signal exceeds a preset threshold, the dynamic range control module automatically reduces its gain. This can effectively suppress sudden excessive volume and prevent the signal amplitude from exceeding the maximum range that the speaker hardware can withstand, thereby avoiding clipping distortion.

[0166] (2) Limiting: Set an absolute upper limit for the signal amplitude. Any signal that attempts to exceed this limit will be forced back to the upper limit. This is the most effective way to prevent signal overload and protect the speaker hardware.

[0167] (3) Gain Compensation: After compressing a signal, its overall loudness may decrease. The dynamic range control module can apply a fixed gain to the processed signal to compensate for the loss of loudness and ensure that the final output volume is moderate.

[0168] For example, you can set a compression threshold of -6dB, a compression ratio of 2:1, and a hard limiter of -0.1dB. In this way, when the signal amplitude exceeds -6dB, the gain of the excess portion will be halved; at the same time, the peak value of the signal will never exceed -0.1dB, ensuring absolute safety of the output.

[0169] The speaker voice enhancement method provided by this invention dynamically adjusts the amplitude of the target audio signal by adding dynamic range control before output. This effectively manages the dynamics of the output signal, preventing sound quality degradation and the risk of howling caused by excessive volume. It also suppresses noise during silent periods, improving the listening experience. This ensures that the sound ultimately played back from the speaker is both clear and stable, while remaining within the hardware's safe operating range, greatly enhancing the reliability and practicality of the entire system.

[0170] Figure 7 Schematic diagram of the structure of the speaker voice enhancement device provided by the present invention. Figure 7 As shown, mainly including but not limited to: The signal acquisition unit 710 is configured to acquire an initial spectrum signal corresponding to the initial audio signal and determine an initial power spectrum signal corresponding to the initial spectrum signal; A signal processing unit 720 is configured to input the initial power spectrum signal into a feedback cancellation network model to obtain a target frequency domain signal output by the feedback cancellation network model, wherein the feedback cancellation network model includes a first sub-model and a second sub-model; The signal generating unit 730 is configured to obtain a target audio signal based on the target frequency domain signal.

[0171] It should be noted that the speaker voice enhancement device provided by the present invention can execute the speaker voice enhancement method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0172] The speaker speech enhancement device provided by the present invention first uses a real signal mask to mask and adjust the frequency domain amplitude of the initial spectrum signal to generate a frequency domain estimation signal, and then uses a complex signal mask to mask and adjust the frequency domain amplitude and phase of the initial spectrum signal to obtain a frequency domain enhancement signal. Finally, the target frequency domain signal is determined based on the frequency domain estimation signal and the frequency domain enhancement signal, so that the output target frequency domain signal has higher fidelity in both amplitude and phase, can remove ambient noise and speaker feedback signals to the greatest extent, and effectively suppress howling.

[0173] Figure 8 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8 As shown, the electronic device may include: a processor 810 , a communication interface 820 , a memory 830 and a communication bus 840 , wherein the processor 810 , the communication interface 820 and the memory 830 communicate with each other via the communication bus 840 . The processor 810 can call the logic instructions in the memory 830 to execute the speaker speech enhancement method, which includes: obtaining an initial spectrum signal corresponding to the initial audio signal, and determining an initial power spectrum signal corresponding to the initial spectrum signal; inputting the initial power spectrum signal into the feedback elimination network model to obtain the target frequency domain signal output by the feedback elimination network model, and the feedback elimination network model includes a first sub-model and a second sub-model; obtaining a target audio signal based on the target frequency domain signal; the first sub-model outputs a real signal mask according to the input initial power spectrum signal; the real signal mask is used to mask the frequency domain amplitude of the initial spectrum signal to obtain a frequency domain estimation signal; the second sub-model receives a three-dimensional spectrum signal obtained by splicing the frequency domain estimation signal and the initial spectrum signal, and outputs a complex signal mask; the complex signal mask is used to mask the frequency domain amplitude and phase of the initial spectrum signal to obtain a frequency domain enhancement signal; the target frequency domain signal is determined based on the frequency domain estimation signal and the frequency domain enhancement signal.

[0174] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0175] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the speaker speech enhancement method provided by the above-mentioned embodiments, the method including: obtaining an initial spectrum signal corresponding to the initial audio signal, and determining an initial power spectrum signal corresponding to the initial spectrum signal; inputting the initial power spectrum signal into a feedback elimination network model, obtaining a target frequency domain signal output by the feedback elimination network model, and the feedback elimination network model including a first sub-model and a second sub-model; a target audio signal is obtained based on the target frequency domain signal; the first sub-model outputs a real signal mask according to the input initial power spectrum signal; the real signal mask is used to mask the frequency domain amplitude of the initial spectrum signal to obtain a frequency domain estimation signal; the second sub-model receives a three-dimensional spectrum signal obtained by splicing the frequency domain estimation signal and the initial spectrum signal, and outputs a complex signal mask; the complex signal mask is used to mask the frequency domain amplitude and phase of the initial spectrum signal to obtain a frequency domain enhancement signal; the target frequency domain signal is determined based on the frequency domain estimation signal and the frequency domain enhancement signal.

[0176] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speaker speech enhancement method provided in the above-mentioned embodiments, the method comprising: obtaining an initial spectrum signal corresponding to an initial audio signal, and determining an initial power spectrum signal corresponding to the initial spectrum signal; inputting the initial power spectrum signal into a feedback elimination network model, obtaining a target frequency domain signal output by the feedback elimination network model, the feedback elimination network model comprising a first sub-model and a second sub-model; obtaining a target audio signal based on the target frequency domain signal; the first sub-model outputs a real signal mask based on the input initial power spectrum signal; the real signal mask is used to mask the frequency domain amplitude of the initial spectrum signal to obtain a frequency domain estimation signal; the second sub-model receives a three-dimensional spectrum signal obtained by splicing the frequency domain estimation signal and the initial spectrum signal, and outputs a complex signal mask; the complex signal mask is used to mask the frequency domain amplitude and phase of the initial spectrum signal to obtain a frequency domain enhancement signal; the target frequency domain signal is determined based on the frequency domain estimation signal and the frequency domain enhancement signal.

[0177] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0178] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for enhancing speaker speech, characterized in that: include: Obtaining an initial frequency spectrum signal corresponding to the initial audio signal, and determining an initial power spectrum signal corresponding to the initial frequency spectrum signal; Inputting the initial power spectrum signal into a feedback cancellation network model to obtain a target frequency domain signal output by the feedback cancellation network model, wherein the feedback cancellation network model includes a first sub-model and a second sub-model; Obtaining a target audio signal based on the target frequency domain signal; The first sub-model outputs a real signal mask according to the input initial power spectrum signal; the real signal mask is used to perform mask adjustment on the frequency domain amplitude of the initial spectrum signal to obtain a frequency domain estimation signal; The second sub-model receives a three-dimensional spectrum signal obtained by splicing the frequency domain estimation signal and the initial spectrum signal, and outputs a complex signal mask; the complex signal mask is used to perform mask adjustment on the frequency domain amplitude and phase of the initial spectrum signal to obtain a frequency domain enhanced signal; The target frequency domain signal is determined based on the frequency domain estimation signal and the frequency domain enhanced signal.

2. The speaker speech enhancement method according to claim 1, wherein: The first sub-model includes a first input module, a first GRU module, a second GRU module, a splicing module, a third GRU module and a first output module; The first input module receives the initial power spectrum signal and inputs the initial power spectrum signal into the first GRU module and the second GRU module in parallel; The first GRU module performs time series modeling on the initial power spectrum signal to obtain a first time series feature; The second GRU module performs time series modeling on the initial power spectrum signal to obtain a second time series feature; The splicing module splices the first time series feature and the second time series feature along the feature dimension to obtain a fused time series feature; The third GRU module receives the fused time series features to perform time series dependency modeling on the fused time series features to obtain final time series features; The first output module is used to map the final timing feature into the real signal mask.

3. The speaker voice enhancement method according to claim 1, wherein: The second sub-model includes a second input module, an encoder module, a bottleneck processing module, a decoder module and a second output module; The second input module inputs the received three-dimensional spectrum signal into the encoder module; The encoder module includes a plurality of cascaded upsampling convolution layers, which are used to perform convolution and upsampling on the three-dimensional spectrum signal layer by layer and output intermediate coding features at each layer; The bottleneck processing module performs temporal context modeling on the intermediate coding features output by the last upsampling convolutional layer to perform filtering and weighting processing of adjacent time frames and output a bottleneck feature after temporal enhancement; The decoder module includes a plurality of cascaded downsampling convolution layers, which are used to downsample and convolve the bottleneck features layer by layer and output a frequency domain decoded signal; The second output module performs convolution mapping on the frequency domain decoded signal and outputs the complex signal mask.

4. The speaker voice enhancement method according to claim 3, characterized in that: The bottleneck processing module includes a gated recurrent unit, a bidirectional gated recurrent unit and a linear mapping layer; The gated recurrent unit receives the intermediate coding feature output by the last upsampling convolutional layer of the encoder module, performs time series modeling on the intermediate coding feature along the time frame dimension and outputs a first time series feature; The bidirectional gated recurrent unit receives the first time series feature, performs bidirectional time series modeling to perform filtering and weighting processing of adjacent time frames, and outputs a second time series feature; The linear mapping layer performs mapping transformation on the second time series feature and outputs the bottleneck feature after time series enhancement.

5. The speaker speech enhancement method according to claim 3, wherein: The downsampling convolutional layers in the decoder module correspond one-to-one to the upsampling convolutional layers in the encoder module; In the process of downsampling and convolving the bottleneck features layer by layer in the decoder module and outputting the frequency domain decoded signal, the intermediate coding features with the same resolution output by the corresponding upsampling convolution layer are fused through jump connections in each downsampling convolution layer.

6. The speaker speech enhancement method according to claim 3, characterized in that: The feedback elimination network model also includes a weighted fusion module; The weighted fusion module performs weighted fusion on the frequency domain estimation signal output by the first sub-model and the frequency domain enhanced signal output by the second sub-model to generate the target frequency domain signal.

7. The speaker speech enhancement method according to any one of claims 1 to 6, characterized in that: The initial audio signal is obtained based on the following method: Get the microphone receiving signal; The microphone received signal is sampled according to a preset synthesis window length and a preset frame shift to obtain the initial audio signal obtained by each sampling.

8. The speaker speech enhancement method according to claim 7, characterized in that: The microphone receiving signal at least includes a target speech signal and a loudspeaker feedback signal; After acquiring the microphone reception signal, eliminating the speaker feedback signal based on an adaptive feedback control algorithm specifically includes: Initializing an adaptive filter, wherein an error signal of the adaptive filter is set to a difference between the microphone reception signal and an estimated signal that predicts the loudspeaker feedback signal; updating the coefficients of the adaptive filter based on a preset adaptive algorithm to determine a minimized error signal; The minimized error signal is used as a new microphone reception signal after eliminating the loudspeaker feedback signal.

9. The speaker speech enhancement method according to claim 8, characterized in that: Before eliminating the loudspeaker feedback signal based on the adaptive feedback control algorithm, the method further includes: The microphone receiving signal is input into a hardware equalizer, so as to use the hardware equalizer to perform gain adjustment on an abnormal frequency band in the microphone receiving signal.

10. The speaker speech enhancement method according to any one of claims 1 to 6, characterized in that: After obtaining the target audio signal based on the target frequency domain signal, the method further includes: performing frequency shift and phase modulation processing and / or half-wave rectification processing on the target audio signal; The frequency shift and phase modulation processing includes increasing or decreasing the spectrum of the target audio signal as a whole by a preset Hz, and periodically changing the phase of the target audio signal; The half-wave rectification process includes converting all sample values ​​smaller than zero in the time domain waveform of the target audio signal into sample values ​​greater than zero by taking absolute values.

11. The speaker voice enhancement method according to claim 10, characterized in that: After performing frequency shift and phase modulation processing and / or half-wave rectification processing on the target audio signal, the method further includes: The amplitude of the target audio signal is dynamically adjusted to control the dynamic range of the loudspeaker output signal.

12. A speaker voice enhancement device, characterized in that: include: a signal acquisition unit, configured to acquire an initial spectrum signal corresponding to the initial audio signal, and determine an initial power spectrum signal corresponding to the initial spectrum signal; a signal processing unit, configured to input the initial power spectrum signal into a feedback cancellation network model to obtain a target frequency domain signal output by the feedback cancellation network model, wherein the feedback cancellation network model includes a first sub-model and a second sub-model; A signal generating unit is configured to obtain a target audio signal based on the target frequency domain signal.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the speaker speech enhancement method according to any one of claims 1 to 11 is implemented.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speaker speech enhancement method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the speaker speech enhancement method according to any one of claims 1 to 11 is implemented.

Images