A low-latency sound reinforcement system acoustic feedback control method, device and system

Abstract

A low-latency sound reinforcement system acoustic feedback control method, comprising: constructing a sound reinforcement system model, establishing a closed-loop transfer function, and determining the system stability condition; constructing an acoustic feedback path and generating an open-loop training data set to simulate the acoustic feedback signal under a critical unstable state; extracting the time-frequency features of the input signal and designing the complex spectrum mapping target of the neural network; constructing a full-subband grouping long-short term memory network, including an encoder, a full-band and sub-band processing module, and a decoder, training the network using a hybrid amplitude spectrum and complex spectrum loss function, optimizing the signal reconstruction strategy to reduce the latency; fine-tuning the network in the simulated closed-loop system, adapting to the actual acoustic environment and nonlinear distortion, and limiting the output signal amplitude through hard clipping; deploying the fine-tuned model to the real sound reinforcement system and cooperating with the frequency shift method to suppress acoustic feedback. The method can effectively suppress acoustic feedback at a lower latency, improve the quality and intelligibility of speech, and increase the maximum stable gain of the sound reinforcement system.

Description

Technical Field

[0001] This invention relates to the field of sound reinforcement technology, and in particular to a method, apparatus and system for sound feedback control of a low-delay sound reinforcement system. Background Technology

[0002] Sound reinforcement systems are used to amplify sound, with typical applications including multimedia classrooms and local conferencing systems. However, in sound reinforcement systems, when the gain exceeds the maximum stable gain, serious acoustic feedback problems arise, severely degrading the quality of the output speech and even causing the system to collapse. Therefore, acoustic feedback control has become an indispensable part of sound reinforcement systems, improving both their sound reinforcement performance and ensuring their stability and safety. Some known acoustic feedback control methods (such as adaptive filtering) require modeling the acoustic feedback path, provide very limited additional stable gain, and struggle to effectively suppress the influence of environmental noise, exhibiting relatively poor robustness. Summary of the Invention

[0003] To address the problems existing in the prior art, embodiments of this application provide a method, apparatus, system, computing device, computer storage medium, and product containing a computer program for controlling acoustic feedback in a low-latency sound reinforcement system. This method can effectively suppress acoustic feedback with low latency, improve speech quality and intelligibility, and significantly enhance the maximum stable gain of the sound reinforcement system.

[0004] In a first aspect, embodiments of this application provide a low-latency sound reinforcement system acoustic feedback control method, comprising: constructing a sound reinforcement system model based on the acquired input speech signal of the sound reinforcement system, the sound signal picked up by the microphone, and the output signal of the loudspeaker; establishing a closed-loop transfer function and determining the system stability conditions; constructing an acoustic feedback path and generating an open-loop training dataset to simulate the acoustic feedback signal under critical instability; extracting the time-frequency features of the input signal and designing a complex spectrum mapping target for the neural network; constructing a full-subband grouped long short-term memory network, including an encoder, full-band and sub-band processing modules, and a decoder, wherein the full-band module models the global frequency domain dependence through grouped LSTM, and the sub-band module processes local time-frequency features through frequency downsampling and independent LSTM; replacing deconvolution operations with linear layers and overlapping addition to reduce computational complexity; training the network with a hybrid amplitude spectrum and complex spectrum loss function to optimize the signal reconstruction strategy to reduce latency; fine-tuning the network in a simulated closed-loop system to adapt to the actual acoustic environment and nonlinear distortion, and limiting the output signal amplitude through hard pruning; and deploying the fine-tuned model to a real sound reinforcement system to suppress acoustic feedback in conjunction with the frequency shift method.

[0005] In some possible implementations, the sound reinforcement system modeling step involves determining the maximum stable gain using the Nyquist stability criterion and simulating the room impulse response based on the virtual source method to construct a feedback path.

[0006] In some possible implementations, generating the open-loop training dataset includes: setting the amplifier gain to within the range of 0.5 to 0.999 times the maximum stable gain; applying frequency shift processing to the microphone signal with feedback to generate input data; and using the pure amplifier output signal without feedback as the training target.

[0007] In some possible implementations, the optimized signal reconstruction strategy to reduce latency includes: using a short-time Fourier transform with a frame shift of 2ms; designing a synthesis window function that relies only on the current frame and the next frame data for signal reconstruction; and controlling the overall algorithm latency to within 4ms.

[0008] In some possible implementations, the method of training the network using a hybrid amplitude spectrum and complex spectrum loss function includes: combining the mean square error of the amplitude spectrum and the real / imaginary part error of the complex spectrum in the hybrid loss function, and eliminating the spectral leakage effect of the window function by recalculating the time spectrum.

[0009] In some possible implementations, the mixed amplitude spectrum and complex spectrum loss function is expressed as:

[0010]

[0011] In the formula, Characterizing the total loss function, Characterizing the mean square error of the amplitude spectrum, Characterizing the mean square error of the complex spectrum The network predicts the time-frequency domain signal, and S represents the target time-frequency domain signal. r and· i Let ||·|| represent the real and imaginary parts of the time-frequency domain feature, respectively. F Characterizes the F-norm of the matrix.

[0012] In some possible implementations, the construction of the sound reinforcement system model includes: determining the maximum stable gain using the Nyquist stability criterion and simulating the room impulse response based on the virtual source method to construct a feedback path.

[0013] In some possible implementations, when the method is deployed, the output of the full-subband grouped long short-term memory network is combined with an adaptive filtering method to form a multi-level acoustic feedback suppression system.

[0014] Secondly, embodiments of this application provide a low-latency sound reinforcement system acoustic feedback control device, comprising: a construction module, used to construct a sound reinforcement system model, establish a closed-loop transfer function, and determine system stability conditions based on the acquired input speech signal of the sound reinforcement system, the acoustic signal picked up by the microphone, and the output signal of the loudspeaker; the construction module is further used to construct an acoustic feedback path and generate an open-loop training dataset to simulate the acoustic feedback signal under critical unstable conditions; a processing module, used to extract the time-frequency features of the input signal and design the complex spectrum mapping target of the neural network; the construction module is further used to construct a full-subband grouped long short-time memory network, including an encoder, a full-band and a subband. The processing module and decoder consist of a full-band module that models global frequency domain dependencies using grouped LSTM, and a sub-band module that processes local time-frequency features using frequency downsampling and independent LSTM. Linear layers and overlapping addition are used instead of deconvolution operations to reduce computational complexity. The processing module also trains the network using a hybrid amplitude spectrum and complex spectrum loss function to optimize the signal reconstruction strategy and reduce latency. Furthermore, the processing module fine-tunes the network in a simulated closed-loop system to adapt to the actual acoustic environment and nonlinear distortion, limiting the output signal amplitude through hard pruning. Finally, the processing module deploys the fine-tuned model to a real sound reinforcement system, working in conjunction with frequency shifting to suppress acoustic feedback.

[0015] Thirdly, embodiments of this application provide a sound reinforcement system, characterized in that it is equipped with an acoustic feedback control device as described in the second aspect.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium including computer-readable instructions that, when read and executed by a computer, cause the computer to perform the method as described in any of the first aspects.

[0017] Fifthly, embodiments of this application provide a computing device including a processor and a memory, wherein the memory stores computer program instructions, which, when executed by the processor, perform the method as described in any of the first aspects.

[0018] Sixthly, embodiments of this application provide a product comprising a computer program that, when the computer program product is run on a processor, causes the processor to perform the method as described in any of the first aspects. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a sound reinforcement system architecture provided in an embodiment of this application;

[0021] Figure 2 This is a schematic flowchart of a low-delay sound reinforcement system acoustic feedback control method provided in an embodiment of this application;

[0022] Figure 3 This is an abstract representation diagram of a sound reinforcement system model provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of an acoustic feedback signal simulation provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of a full-subband grouped long short-term memory network provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of closed-loop training of a full-subband grouped long short-term memory network provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of adding a trained model to a closed-loop system to suppress acoustic feedback, provided in an embodiment of this application.

[0027] Figure 8 This is a schematic diagram of the structure of a low-delay sound reinforcement system acoustic feedback control device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0030] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0031] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0032] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0033] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0034] Figure 1 A schematic diagram of a sound reinforcement system architecture provided in an embodiment of this application is shown. Figure 1 As shown, a sound reinforcement system typically includes a microphone, an amplifier, and a loudspeaker. The microphone receives the sound signal, performs electroacoustic conversion, and conducts necessary signal processing. The amplifier then amplifies the sound signal, and the loudspeaker performs electroacoustic conversion before outputting it. During operation, because the microphone and loudspeaker are in the same sound field, the sound signal played by the loudspeaker is inevitably picked up, amplified, and output again by the microphone. This creates a closed-loop system between the microphone and loudspeaker, resulting in acoustic feedback problems. Although existing deep learning-based acoustic feedback control methods can provide greater stable gain and effectively suppress the influence of environmental noise, combining them with frequency shifting methods can produce severe speech distortion.

[0035] In view of this, embodiments of this application provide a two-stage low-latency acoustic feedback control method for a sound reinforcement system, which uses a deep neural network as a post-processing network for the traditional frequency-shifting method to jointly control the acoustic feedback of the sound reinforcement system. For an indoor sound reinforcement system, firstly, a large number of acoustic feedback paths are constructed using simulated room impulse responses, and the maximum stable gain of the simulated system is calculated based on the Nyquist stability criterion; then, a large amount of data with critical acoustic feedback characteristics processed by the frequency-shifting method is generated in the closed-loop system for offline open-loop training of the deep neural network; subsequently, a low-latency strategy is designed to ensure that the network latency is controlled within 4ms; in addition, a novel loss function is introduced to further improve the performance of the algorithm; finally, the trained network is fine-tuned in the closed-loop system to further improve the network's acoustic feedback suppression capability.

[0036] For example, Figure 2A schematic flowchart of a low-delay sound reinforcement system acoustic feedback control method provided in an embodiment of this application is shown. Figure 2 As shown, the acoustic feedback control method may include the following steps:

[0037] S21: Based on the input speech signal of the sound reinforcement system, the sound signal picked up by the microphone, and the output signal of the loudspeaker, construct a sound reinforcement system model, establish a closed-loop transfer function, and determine the system stability conditions.

[0038] In this embodiment, a mathematical model of a closed-loop sound reinforcement system is established. The model needs to include several key components: the input signal, the sound output from the speaker, the sound received by the microphone, and the path of sound returning from the speaker to the microphone (i.e., the feedback path). The model considers both time-domain and frequency-domain representations. In the time domain, the sound is represented by discrete time point signal values, focusing primarily on how the signal changes over time. In the frequency domain, a short-time Fourier transform is used to decompose the signal into different frequency components, making it clearer which frequencies are prone to feedback. By establishing the mathematical model, the conditions for system stability, i.e., the conditions for preventing feedback, can be derived. Specifically, feedback occurs when the system's open-loop gain (the amplification factor for one sound cycle) simultaneously satisfies both amplitude and phase conditions at certain frequencies.

[0039] For a detailed abstract representation of the sound reinforcement system model, please refer to [reference needed]. Figure 3 Let v(n) be the input speech signal, y(n) be the sound signal picked up by the microphone, u(n) be the loudspeaker output signal, f(n) be the feedback path transfer function, and g(n) be the transfer function of the sound reinforcement system amplifier. Then, we have the following expression:

[0040] y(n) = v(n) + u(n) * f(n)

[0041] u(n) = y(n) * g(n)

[0042] In the formula, * represents the convolution operation, n represents the number of discrete time points, and n is a natural number, i.e., n = [0, 1, ..., n]. Performing a Short-Time Fourier Transform (STFT) on the time-domain signal, the time-frequency domain expression of the above formula is:

[0043] Y(ω,l)=V(ω,l)+U(ω,l)F(ω,l)

[0044] U(ω,l)=Y(ω,l)G(ω,l)

[0045] In the formula, Y(ω,l) represents the time-frequency domain characteristics of the sound signal picked up by the microphone, V(ω,l) represents the time-frequency domain characteristics of the input speech signal, U(ω,l) represents the time-frequency domain characteristics of the loudspeaker output signal, F(ω,l) represents the time-frequency domain characteristics of the feedback path transfer function, G(ω,l) represents the time-frequency domain characteristics of the amplifier transfer function of the sound reinforcement system, ω represents the number of frequency points and ω is a natural number, and l represents the number of frames and l is a natural number.

[0046] Combining the above equations, we can obtain the closed-loop transfer function from the microphone to the loudspeaker, expressed as:

[0047]

[0048] In the formula, G(ω,l)F(ω,l) represents the open-loop transfer function.

[0049] Typically, assuming the transfer function of the feedback path is fixed or slowly varying, then F(ω,l)≈F(ω). A sound reinforcement system is stable only when all poles of the closed-loop system lie within the unit circle; the appearance of acoustic feedback is a manifestation of instability in a closed-loop sound reinforcement system. According to the Nyquist stability criterion, a closed-loop system is unstable when the open-loop transfer function satisfies the following condition:

[0050]

[0051] In the formula, Represents the set of integers, |·| represents the modulo value, and ∠· represents the phase.

[0052] When the phase of the open-loop transfer function is an integer multiple of 2π and the magnitude of the transfer function is greater than 1, the closed-loop sound reinforcement system is unstable at the corresponding frequency, resulting in howling. If we assume the amplifier only amplifies, i.e., G(ω,l)=G, where G is the amplification factor, then the maximum stable gain G that keeps the closed-loop sound reinforcement system stable can be calculated. max for:

[0053]

[0054] In the formula, Ω(l) = {ω|∠F(ω,l) = 2kπ}, which is the set of frequencies that satisfy the corresponding phase condition. In practical applications, the gain of the sound reinforcement system must be less than the maximum closed-loop stable gain. Without causing ambiguity, the frame index l and frequency index ω will be omitted in the subsequent description of the scheme.

[0055] S22: Construct an acoustic feedback path and generate an open-loop training dataset to simulate the acoustic feedback signal under critical instability.

[0056] In this embodiment, the first step is to simulate the path of sound from the speaker back to the microphone, i.e., the feedback path. The simulation must consider the acoustic characteristics of the room, such as wall reflections, and typically uses the virtual source method to calculate the room's impulse response. With this feedback path model, the system's behavior at different gains can be simulated, especially near the critical steady state. Next, a large amount of training data needs to be generated. This process involves generating a clean speech signal as input, then passing this signal through a simulated closed-loop sound reinforcement system where the amplifier gain is intentionally set close to an unstable range, thus producing a signal with feedback tendency. These signals are then processed using a frequency-shifting method, which suppresses feedback but introduces distortion. The frequency-shifted signal serves as the input data for the neural network, while the corresponding target data is the ideal output signal without feedback loop. This constitutes the data pair required for open-loop training: the input is the frequency-shifted signal with residual issues, and the output is the ideal signal that should be obtained. This data generation process can be performed on a large scale, providing sufficient samples for subsequent deep learning training.

[0057] Specifically, as shown in step S21, when the system does not satisfy the Nyquist stability criterion, the signal at frequency ω will be continuously amplified, forming a howling effect. In an indoor sound reinforcement system, the transfer function of the feedback path is the sum of all room impulse responses from the microphone to the loudspeaker. In this embodiment, the virtual source method is used to simulate the room impulse response, after which the maximum stable gain G can be obtained. max . Figure 4 This illustration shows a schematic diagram of an acoustic feedback signal simulation provided in an embodiment of this application. The signal simulation is related to... Figure 3 The difference lies in the addition of a frequency shifter after the amplifier transfer function in the sound reinforcement system. To generate a speech signal with acoustic feedback, a method such as... Figure 4 The acoustic feedback signal simulation system shown generates an analog acoustic feedback signal, where the amplifier gain G ∈ [0.5G]. max 0.999G max This is to put the system in a state prone to howling. Frequency shifting is a traditional acoustic feedback control method that maintains stability in a closed-loop sound reinforcement system by controlling the phase condition of the Nyquist stability criterion. Specifically, it controls the phase of the microphone-received signal so that after one cycle of the closed-loop sound reinforcement system, the frequency components of the acoustic feedback signal have different phases each time they reach the microphone. Therefore, it can significantly improve the maximum stable gain of the closed-loop sound reinforcement system. However, frequency shifting does not cut off the acoustic feedback path, and this method severely degrades the quality of the output speech. Therefore, this embodiment uses a deep neural network as the post-processing network for the traditional frequency shifting method to jointly control the acoustic feedback of the sound reinforcement system. Frequency shifting can be implemented by constructing an analytic signal of the microphone-received signal y(n). The analytic signal of y(n) can be obtained by designing a Hilbert filter:

[0058]

[0059] In the formula, The Hilbert transform characterizes the signal received by the microphone, and j represents the imaginary unit.

[0060] Since the Hilbert filter is non-causal, it introduces a time delay equal to half the filter length. In this embodiment, the Hilbert filter length is 33. The output signal of the frequency shifter is then obtained as follows:

[0061]

[0062] In the formula, The real part is represented by H(ω,n), which represents the frequency response of the Hilbert filter. m f is calculated based on the frequency shift. m Characterizes the frequency shift, and has f m =ω m (f s / 2π), f s The sampling frequency is represented. In this embodiment, the frequency shift can be 5Hz.

[0063] After frequency shifting, a large amount of data with critical acoustic feedback characteristics can be obtained, which can be used for offline open-loop training of deep neural networks (DNNs). Through the deep neural network, the feedback-free signal s(n) can be directly estimated.

[0064] s(n)=Gv(t-τf s )

[0065] In the formula, τ represents the time delay of the sound reinforcement system, and t represents the time.

[0066] A large number of data pairs (d(n), s(n)) constitute an open-loop dataset, where d(n) is the feedback signal after frequency shifting, which serves as the input to the DNN, and s(n) is the signal that should be output without feedback, i.e., without feedback influence, which is the target that the DNN needs to predict.

[0067] S23: Extract the time-frequency features of the input signal and design the complex spectrum mapping target of the neural network.

[0068] In this embodiment, time-frequency analysis is performed on the sound signal processed by the frequency shifting method. A short-time Fourier transform is used to decompose the sound into representations of different time segments and frequency components, obtaining a complex spectrum containing real and imaginary parts. This spectrum data is reorganized into a format suitable for neural network input, separating the real and imaginary parts into two channels, and then arranging them into a three-dimensional tensor according to the time and frequency dimensions. Next, the learning objective of the neural network is designed; here, it is chosen to allow the network to directly predict the time-frequency representation of the sound signal that the speaker should output under ideal conditions. To enable better learning, the spectral amplitude is compressed, thus balancing the importance of different frequency components. The network needs to simultaneously predict the amplitude and phase information of the output signal, as both aspects are crucial to sound quality. This design ensures that the neural network not only eliminates the feedback components remaining from the frequency shifting method but also repairs the sound quality damage caused by the frequency shifting method, ultimately outputting a clear and natural sound signal.

[0069] Specifically, during feature extraction, a short-time Fourier transform (STFT) is performed on the frequency-shifting output signal d(n) to obtain a complex time-frequency spectrum. The real and imaginary parts of the complex spectrum are concatenated along the channel dimension to form a tensor of dimension 2xTxF, where T is the total number of frames in the input speech and F is the total number of frequency points. This tensor serves as the input to the neural network. The goal of the DNN is to directly predict the time-frequency spectrum of the target signal. A complex spectrum mapping method is used, i.e.

[0070]

[0071]

[0072] in,· r and· i These represent the real and imaginary parts of the time-frequency domain characteristics, respectively. for The time-frequency domain characteristics, This is the network prediction value. Characterizing the complex spectral mapping, D c The frequency-domain characteristics of the frequency-shift method output signal after amplitude spectrum compression are represented by Φ, where Φ represents the parameters of the DNN model, and D is the input feature. For the predicted value, β c This is the amplitude spectrum compression factor, typically 0.5.

[0073] In some more specific embodiments, the audio signal is sampled at a frequency of 16 kHz, each frame is 20 ms long, the frame shift is 2 ms, the number of Fourier transform points is 320, and the total number of frequency points is 161.

[0074] S24: Construct a full-subband grouped long short-term memory network, including an encoder, full-band and subband processing modules, and a decoder.

[0075] In this embodiment, a deep neural network structure specifically designed to handle acoustic feedback problems is presented. This neural network employs an architecture combining global and local information processing to capture full-band and sub-band information. It mainly consists of three parts: an encoder responsible for converting input features into a high-dimensional representation, a core processing module for learning sound features, and a decoder for converting the processed features back to the time-frequency domain. The encoder uses two-dimensional convolutional layers to extract preliminary features. The core processing module combines two different long short-term memory networks: one responsible for capturing global information across the entire frequency band, and the other focused on processing the details of each sub-band. To reduce computational complexity, a grouping processing strategy is adopted, dividing the features into several groups, processing them separately, and then merging them. The decoding part uses deconvolutional layers to reconstruct the output signal. Low latency is also considered in the network design; operations in the time dimension do not require caching information from past frames, relying only on information from the current and previous frames.

[0076] Specifically, Figure 5 This is a schematic diagram of a full-subband grouped long short-term memory network provided in an embodiment of this application. Please refer to it. Figure 5This embodiment employs a full-subband grouped long short-term memory (FSB-GLSTM) network for acoustic feedback suppression. It consists of three parts: an encoder, a full-subband grouped long short-term memory module, and a decoder. The encoder includes a 1×3 two-dimensional convolutional layer (Conv2D) with kernels of 1×3 along both the time and frequency dimensions, and a normalization layer to extract the D-dimensional embedding for each time-frequency point. The decoder is a corresponding 1×3 two-dimensional deconvolutional layer (Deconv2D) to obtain the real and imaginary parts of the output speech time-frequency spectrum. The full-subband grouped long short-term memory module can be divided into a full-band module and a sub-band module. In the full-band module, for a given D×T×F dimensional input embedding, the T×F embedding for each frame is first compressed into a frame-level embedding based on the Conv2D layer. Then, the grouped LSTM is used to model the frame-level embedding. Finally, the D-dimensional embedding for each time-frequency point is recovered based on the corresponding Deconv2D layer. In this way, the grouped LSTM layer can simultaneously model all frequency components at the current time, thereby capturing full-band information. In this embodiment, a grouped LSTM layer is used to reduce the computational complexity of the model. Specifically, the grouped LSTM layer divides the input embedding along the feature dimension, then models each part using an independent LSTM, and finally concatenates them. This significantly reduces the computational complexity of the model with minimal performance loss. Furthermore, considering that in practice, deconvolution first pads with zeros based on the stride before performing regular convolution operations, when the stride is greater than 1, the zero-padding operation introduces information redundancy, significantly increasing computational complexity. Therefore, this embodiment replaces the Deconv2D layer with a linear layer and overlapping addition method. This reduces the computational complexity to approximately 1 / J of the original Deconv2D layer, where J is the deconvolution stride. Considering that acoustic feedback generally only occurs at certain frequency components, this embodiment also uses a sub-band module to model each frequency component separately to capture its time-frequency information. In addition, to further reduce computational complexity, a Conv2D layer is used to downsample the frequency, reducing the input and output feature dimensions of the sub-band LSTM. Note that in FSB-GLSTM, the kernel size is 1 along the time dimension. Therefore, this embodiment does not require an additional buffer to store information from past frames. It only needs to cache the cell state and hidden state of the previous LSTM to model the time information. As a result, the network requires very little memory during operation and is more suitable for streaming inference.

[0077] In some more specific embodiments, the specific network hyperparameters are described and set as shown in Table 1. In this embodiment, B=6, D=8, E=4, I=8, J=4, H=128, E'=8, I'=5, J'=5, and H'=16 are selected, resulting in a final parameter count of 723.67K and a computational complexity (MACs) of 716.57M / s. Furthermore, during training, the network uses a batch size of 32, 150 iterations, and the Adam optimizer with a learning rate of 0.001. If the validation set loss does not decrease after 3 consecutive iterations, the learning rate is halved; if it does not decrease after 6 consecutive iterations, training is stopped.

[0078] Table 1

[0079]

[0080] S25: The network is trained using a hybrid amplitude spectrum and complex spectrum loss function, and the signal reconstruction strategy is optimized to reduce latency.

[0081] In this embodiment, the neural network needs to learn how to repair the audio signal while ensuring low latency throughout the processing. First, a suitable loss function needs to be selected to guide the neural network's learning direction. This embodiment uses a hybrid loss function combining amplitude spectrum and complex spectrum, requiring the network's output sound to match the target in energy distribution while ensuring the real and imaginary parts are as close to their true values ​​as possible. This guarantees both clarity and naturalness of the speech. To avoid spectral leakage from the analysis window affecting training performance, a synthesis window is used to convert the network's predicted spectrum back to the time domain signal. Then, another analysis window is used to recalculate the spectrum, and finally, the loss is evaluated based on this recalculated spectrum. Regarding low-latency strategies, by optimizing the synthesis window function and using overlapping addition, the system only needs information from the current and next frames to complete the reconstruction, keeping the overall latency very low. Specifically, a small frame shift parameter is chosen, combined with a specific window function design, to compress the algorithm latency to 4 milliseconds while maintaining sufficient frequency resolution.

[0082] Specifically, according to step S24, the output signal of the loudspeaker is obtained by performing an inverse short-time Fourier transform on the time spectrum of the predicted speech output by the network. Since the overlapping addition method introduces a time delay, reducing the window length can further reduce the algorithm's time delay, but this will reduce the frequency resolution of the time-frequency features, thus negatively impacting the acoustic feedback suppression performance. Therefore, this patent employs a smaller synthesis window, reducing the algorithm's time delay to 4ms while maintaining frequency resolution. Specifically, the analysis window selected in this patent is:

[0083]

[0084] In the formula, N is the frame length, R is the frame shift, and t is the number of discrete time points within a frame. During signal reconstruction, the current frame and the next frame are overlapped and added together to obtain the output of the current frame:

[0085]

[0086] Where t = [1,…,R], For each frame of the time-domain signal, n is the frame number and k is the frequency point number. and The synthesis window functions are defined as follows:

[0087]

[0088] At this point, if a frame shift of 2ms is selected, i.e., R = 32, the final network latency is 4ms. In this embodiment, the loss function for network training is selected as the mean square error (MSE) loss function between the amplitude spectrum and complex spectrum of the predicted speech and the target speech, which is:

[0089]

[0090] In the formula, ‖·‖ F This represents the F-norm of a matrix.

[0091] Considering w m The spectral leakage of the (t) window function is quite severe, therefore, the time-domain signal is synthesized first when calculating the loss function. Then it is converted to the time-frequency domain, that is Then, the MSE loss is calculated, where can be any type of window function. In this patent, w is selected. h (t) represents a Hanning window with a frame length of 20ms and a frame shift of 10ms.

[0092] S26: Fine-tune the network in the analog closed-loop system to adapt to the actual acoustic environment and nonlinear distortion.

[0093] This embodiment addresses the adaptability issue of neural networks in real-world application scenarios. Since previous training was conducted in an open-loop environment, while in real-world scenarios the network output is played through a speaker and then picked up again by a microphone, forming a closed-loop feedback, this difference leads to a decrease in model performance. To eliminate this effect, a simulated closed-loop system is needed for fine-tuning training. This simulated system contains a complete signal loop; the input speech is first mixed with the network-processed output signal through the simulated feedback path to form a closed-loop microphone input, which is then preprocessed using a frequency-shifting method before being fed into the neural network. Simultaneously, the simulated system also considers the nonlinear characteristics of real speakers, such as hard limiting of signal amplitude. During fine-tuning, the previous mixing loss function is used, but the training data becomes the signal pairs generated by this closed-loop simulated system.

[0094] Specifically, considering the model mismatch problem when using an open-loop trained network in a real closed-loop sound reinforcement system, this patent fine-tunes the open-loop trained model in a closed-loop simulation system. Figure 6 This diagram illustrates a closed-loop training method for a full-subband grouped long short-term memory network. Figure 4 The difference is that an FSB-GLSTM is connected after the frequency shifter. Please refer to [link / reference]. Figure 6 In the simulated closed-loop system, assuming f(n) is the simulated room impulse response, g(n) only acts as an amplifier, and the delay is τ. g Then we have g(n) = Gδ(n-τ) g f s ), where δ(n) is the unit impulse response. Considering that real loudspeakers all have nonlinear distortion, it is assumed that the analog loudspeaker has a saturation region, that is, the input audio signal is hard-trimmed:

[0095]

[0096] in, The maximum output value of the loudspeaker is set to 1 during the simulation. Therefore, the input signal of the microphone at this time is:

[0097]

[0098] By predicting the loudspeaker output signal in the simulated closed-loop system, the model can be fine-tuned based on the loss function in step S25, further improving the acoustic feedback suppression capability of this patent.

[0099] S27: Deploy the fine-tuned model to a real sound reinforcement system and work with the frequency shift method to suppress acoustic feedback.

[0100] In this embodiment, a fully trained neural network model is deployed into a real sound reinforcement system for practical application. Please refer to... Figure 7 At this point, the entire system begins operating in a closed-loop manner. The audio signal captured by the microphone is first preprocessed using a frequency-shifting method to eliminate most of the feedback, and then fed into a trained neural network for fine processing. The network performs in-depth analysis of the signal, identifying and suppressing residual feedback components that the frequency-shifting method could not completely eliminate, while also repairing speech distortion caused by the frequency-shifting process. The processed signal is then amplified by a power amplifier to drive the speaker for playback, and the played sound is fed back to the microphone through the acoustic environment of the room, forming a complete closed loop. In actual operation, the network needs to process the continuous audio stream in real time, maintaining extremely low processing latency to ensure system synchronization.

[0101] The above describes the acoustic feedback control method for low-latency sound reinforcement systems provided in this application. Addressing the issue of acoustic feedback in sound reinforcement systems, a model is trained using open-loop data as the training set, then fine-tuned in a closed-loop system. Finally, the trained model is placed in the closed-loop sound reinforcement system to suppress the acoustic feedback signal. In practical applications, for signals with acoustic feedback received by the sound reinforcement system, a frequency shifting method is first applied for preprocessing. Then, the trained network is used to remove residual acoustic feedback components, resulting in an accurate estimate of the speaker output signal. Furthermore, since hearing aid systems and assistive hearing systems also fall under the category of sound reinforcement systems, the method proposed in this application is also applicable to hearing aids and assistive hearing devices. The advantage of this method is that the neural network does not work alone but collaborates with the traditional frequency shifting method. The frequency shifting method provides basic stability, while the neural network further improves sound quality and suppression effects. If needed, it can also be combined with other acoustic feedback suppression techniques, such as using the neural network output as a reference signal for an adaptive filter to form a more powerful hybrid suppression system. The final effect is to achieve a more natural and clear speech output quality than traditional methods while maintaining system stability.

[0102] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Furthermore, in some possible implementations, each step in the above embodiments may be selectively executed according to actual circumstances; it may be partially or fully executed, without limitation here. All or part of any feature of any embodiment of this application can be freely and arbitrarily combined without contradiction. The combined technical solutions are also within the scope of this application.

[0103] Based on the methods in the above embodiments, this application also provides a low-delay sound reinforcement system acoustic feedback control device. For example, Figure 8 A schematic diagram of the acoustic feedback control device for a low-delay sound reinforcement system provided in this application is shown. Figure 8As shown, the device 800 includes 801 and a processing module 802.

[0104] Among them, the construction module 801 is used to construct a sound reinforcement system model, establish a closed-loop transfer function, and determine the system stability conditions based on the input speech signal of the sound reinforcement system, the sound signal picked up by the microphone, and the output signal of the loudspeaker.

[0105] The construction module 801 is also used to construct an acoustic feedback path and generate an open-loop training dataset to simulate the acoustic feedback signal under critical instability.

[0106] Processing module 802 is used to extract the time-frequency features of the input signal and design the complex spectrum mapping target of the neural network;

[0107] The construction module 801 is also used to construct a full-subband grouped long short-term memory network, which includes an encoder, a full-band and subband processing module and a decoder. The full-band module models the global frequency domain dependency through grouped LSTM, and the subband module processes local time-frequency features through frequency downsampling and independent LSTM. Linear layers and overlapping addition are used to replace deconvolution operations to reduce computational complexity.

[0108] The processing module 802 is also used to train the network using a hybrid amplitude spectrum and complex spectrum loss function, and optimize the signal reconstruction strategy to reduce latency;

[0109] The processing module 802 is also used to fine-tune the network in the analog closed-loop system to adapt to the actual acoustic environment and nonlinear distortion, and to limit the output signal amplitude through hard clipping.

[0110] The processing module 802 is also used to deploy the fine-tuned model to a real sound reinforcement system and work with the frequency shift method to suppress acoustic feedback.

[0111] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0112] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0113] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0114] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0115] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0116] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A method for acoustic feedback control of a low-delay sound reinforcement system, characterized in that, The method includes: Based on the input speech signal of the sound reinforcement system, the sound signal picked up by the microphone, and the output signal of the loudspeaker, a sound reinforcement system model is constructed, a closed-loop transfer function is established, and the system stability conditions are determined. Construct an acoustic feedback path and generate an open-loop training dataset to simulate the acoustic feedback signal under critical instability. Extract the time-frequency features of the input signal and design the complex spectrum mapping target of the neural network; A full-subband grouped long short-term memory network is constructed, which includes an encoder, full-band and subband processing modules, and a decoder. The full-band module models the global frequency domain dependency through grouped LSTM, and the subband module processes local time-frequency features through frequency downsampling and independent LSTM. Linear layers and overlapping addition are used to replace deconvolution operations to reduce computational complexity. A network is trained using a hybrid amplitude spectrum and complex spectrum loss function to optimize the signal reconstruction strategy and reduce latency; the analysis window is as follows: In the formula, N is the frame length, R is the frame shift, and t is the number of discrete time points within a frame. During signal reconstruction, the current frame and the next frame are overlapped and added together to obtain the output of the current frame: ,in, , For each frame of the time-domain signal, n is the frame number and k is the frequency point number. and The synthesis window functions are defined as follows: A frame shift of 2 ms was selected, resulting in a final network latency of 4 ms. The loss function for network training was chosen as the mean square error (MSE) between the amplitude spectrum and complex spectrum of the predicted and target speech, which is: In the formula, Characterizing the total loss function, Characterizing the mean square error of the amplitude spectrum, Characterizing the mean square error of the complex spectrum Characterizes network prediction of time-frequency domain signals, Characterizes the target's time-frequency domain signal. and Let these represent the real and imaginary parts of the time-frequency domain characteristics, respectively. Characterization matrix F norm; In the analog closed-loop system, the network is fine-tuned to adapt to the actual acoustic environment and nonlinear distortion, and the output signal amplitude is limited by hard clipping; The finely tuned model was deployed to a real sound reinforcement system and used in conjunction with the frequency shift method to suppress acoustic feedback.

2. The method according to claim 1, characterized in that, In the modeling steps of the sound reinforcement system, the maximum stable gain is determined by the Nyquist stability criterion, and the room impulse response is simulated based on the virtual source method to construct the feedback path.

3. The method according to claim 1, characterized in that, The generated open-loop training dataset includes: Set the amplifier gain to within the range of 0.5 to 0.999 times the maximum stable gain; Frequency shifting is applied to the microphone signal containing feedback to generate input data; The pure amplifier output signal without feedback is used as the training target.

4. The method according to claim 1, characterized in that, The method of training the network using a hybrid amplitude spectrum and complex spectrum loss function includes: The hybrid loss function combines the mean square error of the amplitude spectrum with the real / imaginary part error of the complex spectrum, and eliminates the spectral leakage effect of the window function by recalculating the time spectrum.

5. The method according to claim 1, characterized in that, The construction of the sound reinforcement system model includes: The maximum stable gain was determined using the Nyquist stability criterion, and the room impulse response was simulated using the virtual source method to construct the feedback path.

6. The method according to claim 1, characterized in that, When the method is deployed, the output of the full-subband grouped long short-term memory network is combined with the adaptive filtering method to form a multi-level acoustic feedback suppression system.

7. A low-delay sound reinforcement system acoustic feedback control device, characterized in that, The device includes: The module is used to construct a sound reinforcement system model, establish a closed-loop transfer function, and determine the system stability conditions based on the input speech signal, the sound signal picked up by the microphone, and the output signal of the loudspeaker. The building module is also used to construct an acoustic feedback path and generate an open-loop training dataset to simulate the acoustic feedback signal under critical instability. The processing module is used to extract the time-frequency features of the input signal and design the complex spectrum mapping target of the neural network; The construction module is also used to construct a full-subband grouped long short-term memory network, which includes an encoder, full-band and subband processing modules and a decoder. The full-band module models the global frequency domain dependency through grouped LSTM, and the subband module processes local time-frequency features through frequency downsampling and independent LSTM. Linear layers and overlapping addition are used to replace deconvolution operations to reduce computational complexity. The processing module is also used to train the network using a hybrid amplitude spectrum and complex spectrum loss function, and to optimize the signal reconstruction strategy to reduce latency; The processing module is also used to fine-tune the network in the analog closed-loop system to adapt to the actual acoustic environment and nonlinear distortion, and to limit the output signal amplitude through hard clipping. The processing module is also used to deploy the fine-tuned model to a real sound reinforcement system, and to work with the frequency shift method to suppress acoustic feedback.

8. A sound reinforcement system, characterized in that, The acoustic feedback control device as described in claim 7 is deployed.

Images