A Smart Suppression Method for Feedback Based on Audio Feature Recognition
By integrating continuous timbre characteristics with closed-loop propagation characteristics in a sound reinforcement system, and utilizing gated loop networks and Kalman filtering techniques, feedback signals are identified and suppressed, thus solving the problem of feedback path state updates in a sound reinforcement system and achieving effective suppression of feedback signals.
Patent Information
- Application Number
- CN202611433056.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-16
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-09-16
AI Technical Summary
In a sound reinforcement system, how can we effectively identify the sources of frequency components in the real-time audio from the microphone and control the recursive update of the feedback path state accordingly to suppress howling signals?
By fusing continuous timbre features with closed-loop propagation features, and utilizing gated loop networks and Kalman filtering techniques, feedback signals are identified and suppressed. Specific steps include acquiring audio data, extracting Mel-frequency cepstral coefficients and fundamental frequency sequences to form timbre trajectory data and closed-loop spectrum data, using the Kalman update sequence to recursively deduce the feedback path state, generating a feedback estimation signal, and subtracting the feedback estimation signal from the real-time microphone audio data.
It achieves effective identification and suppression of howling signals, and provides closed-loop spectrum data through continuous correlation across channels and time frames, ensuring accurate updates of feedback path status and continuous characterization of timbre features.
Smart Images

Figure CN122952241B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of audio signal processing technology for public address systems, specifically to a method for intelligent suppression of howling based on audio feature recognition. Background Technology
[0002] In stage performances, conference sound reinforcement, and live audio processing, the sound captured by the microphone is processed by the mixing equipment and then sent to the main loudspeakers and monitor speakers. The sound signal output from the loudspeakers propagates through the sound field and re-enters the microphone, creating an acoustic feedback path between the microphone, audio processing equipment, and loudspeakers. When certain frequency components accumulate continuously in this feedback path, a howling signal with persistent narrow-band spectral peaks is generated.
[0003] Existing sound reinforcement systems can process acoustic feedback through methods such as spectrum detection, adaptive filtering, and feedback path estimation. In actual performances, the fundamental frequency and harmonic components generated by the actor's continuous vocalization also exhibit continuous spectral peak characteristics. Furthermore, the real-time microphone audio simultaneously contains both the actor's vocal components and microphone feedback components returned from the main amplification and monitor sound fields. Therefore, when performing feedback path estimation, it is necessary to combine audio spectrum characteristics, actor timbre characteristics, and the temporal correlation of the sound reinforcement signal between the microphone, main amplification, and monitor channels to identify frequency components from different sources.
[0004] In summary, how to identify the source of frequency components in real-time microphone audio and control the recursive update of the feedback path state accordingly is a technical problem that urgently needs to be solved in this field.
[0005] To address this, a method for intelligent suppression of howling based on audio feature recognition is proposed. Summary of the Invention
[0006] The purpose of this invention is to provide a method for intelligent suppression of feedback based on audio feature recognition. By fusing continuous timbre features with closed-loop propagation features of the sound reinforcement system, the method achieves the identification and suppression of feedback signals. The method includes: acquiring microphone test, real-time microphone, main reinforcement transmission, and monitor transmission audio data; constructing timbre trajectory data and closed-loop spectrum data; inputting the timbre continuity sequence of candidate frequency components and the closed-loop spectrum data into a gated recurrent network, outputting a Kalman update sequence; recursively calculating the feedback path state band by band based on the Kalman update sequence, generating a main reinforcement feedback estimation signal and a monitor feedback estimation signal; and subtracting the feedback estimation signal from the real-time microphone audio data to obtain the output audio data.
[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for intelligent suppression of howling based on audio feature recognition includes: Acquire the main amplifier audio data, monitor audio data, test audio frame sequence, microphone real-time audio frame sequence, main amplifier audio frame sequence, and monitor audio frame sequence; extract Mel frequency cepstral coefficients and fundamental frequency sequence from the test audio frame sequence to form timbre trajectory data; Spectral peaks are extracted from the real-time microphone audio frame sequence to obtain candidate frequency component sequences. The cyclic relationship of the candidate frequency component sequences in the real-time microphone audio frame sequence, the main amplifier audio frame sequence, and the monitor audio frame sequence is traced to form closed-loop spectral data. Mel frequency cepstral coefficients and fundamental frequency sequences of the microphone real-time audio frames corresponding to the candidate frequency component sequences are extracted, and matched with timbre trajectory data to obtain timbre continuation sequences. The timbre continuation sequences and closed-loop spectrum data are input into a gated recurrent network to output a Kalman update sequence. The feedback estimation results are obtained by recursively deriving from the Kalman update sequence, using the main amplifier transmitted audio frame sequence and the monitor transmitted audio frame sequence as references and the microphone real-time audio frame sequence as observations. The feedback estimation signal is generated based on the feedback estimation result, the main amplifier audio data, and the monitor audio data; the feedback estimation signal is subtracted from the microphone real-time audio data to obtain the output audio data.
[0008] Preferably, the process of acquiring the main speaker transmitted audio data, the monitor transmitted audio data, the test audio frame sequence, the microphone real-time audio frame sequence, the main speaker transmitted audio frame sequence, and the monitor transmitted audio frame sequence includes: Acquire microphone test audio data and microphone real-time audio data, as well as main amplifier transmit audio data and monitor transmit audio data; synchronize the microphone test audio data, microphone real-time audio data, main amplifier transmit audio data, and monitor transmit audio data with the same sampling frequency to obtain synchronized audio data; divide the synchronized audio data into frames with the same frame length and frame shift to obtain test audio frame sequence, microphone real-time audio frame sequence, main amplifier transmit audio frame sequence, and monitor transmit audio frame sequence, respectively.
[0009] Preferably, the process of forming timbre trajectory data includes: Short-time Fourier transform and feature extraction are performed on the audio frame sequence of the test audio to obtain the spectral peak position, spectral peak duration, fundamental frequency sequence and corresponding harmonic sequence, Mel frequency cepstral coefficient and Mel frequency cepstral coefficient change sequence of each audio frame. According to the continuation order of the same fundamental frequency sequence and corresponding harmonic sequence in consecutive audio frames, the spectral peak position, spectral peak duration, fundamental frequency sequence and corresponding harmonic sequence, Mel frequency cepstral coefficient and Mel frequency cepstral coefficient change sequence of adjacent frames are continuously correlated to form timbre trajectory data.
[0010] Preferably, the process of forming closed-loop spectral data includes: Short-time Fourier transforms are performed on the microphone real-time audio frame sequence, the main amplifier transmit audio frame sequence, and the monitor transmit audio frame sequence to obtain the microphone real-time spectrum sequence, the main amplifier transmit spectrum sequence, and the monitor transmit spectrum sequence, respectively. Narrowband spectral peaks that persist in continuous audio frames are extracted from the microphone real-time spectrum sequence to obtain a candidate frequency component sequence. For each candidate frequency component in the candidate frequency component sequence, transmit frequency components with the same frequency evolution order are searched in the main amplifier transmit spectrum sequence and the monitor transmit spectrum sequence, respectively. According to the time sequence of the transmit frequency component entering the microphone real-time spectrum sequence, re-entering the main amplifier transmit spectrum sequence and the monitor transmit spectrum sequence, and returning to the microphone real-time spectrum sequence again, the candidate frequency components are continuously correlated across frames to form closed-loop spectral data.
[0011] Preferably, the process of obtaining the timbre continuity sequence includes: Mel frequency cepstral coefficients, Mel frequency cepstral coefficient variation sequences between adjacent frames, spectral peak duration relationships, fundamental frequency sequences, and corresponding harmonic sequences are extracted from the microphone real-time audio frame sequences corresponding to the candidate frequency component sequences. The Mel frequency cepstral coefficients, Mel frequency cepstral coefficient variation sequences between adjacent frames, spectral peak duration relationships, fundamental frequency sequences, and corresponding harmonic sequences are then continuously matched with the timbre trajectory data to obtain the timbre duration sequences corresponding to each candidate frequency component.
[0012] Preferably, the process of outputting the Kalman update sequence includes: Feature masks for corresponding candidate frequency components are generated based on the closed-loop spectrum data; the feature masks are used to filter the timbre continuation sequence to obtain a restricted timbre sequence; the restricted timbre sequence and the closed-loop spectrum data are input into a gated recurrent network in time frame order, and the gated recurrent network outputs the actor source state, feedback source state, and mixed source state corresponding to each candidate frequency component; the candidate frequency components corresponding to the actor source state and their harmonic components with the same fundamental frequency relationship are marked as held states, the candidate frequency components corresponding to the feedback source state are marked as updated states, and the candidate frequency components corresponding to the mixed source state are marked as continued current states; the Kalman update sequence is generated according to the marked states and corresponding frequency positions of each candidate frequency component.
[0013] Preferably, the process of recursively obtaining the feedback estimation result from the Kalman update sequence includes: The main amplifier transmit audio frame sequence and the monitor transmit audio frame sequence are used as reference audio data, and the microphone real-time audio frame sequence is used as the observed audio data. Kalman feedback path state recursion is performed band by band according to the Kalman update sequence. For frequency bands marked as "hold state", the feedback path state corresponding to the previous audio frame is maintained. For frequency bands marked as "update state", the feedback path state is updated according to the reference audio data and observed audio data corresponding to the current audio frame. For frequency bands marked as "continue current state", the feedback path state corresponding to the previous audio frame is used and the state determination is continued in the next audio frame. The feedback estimation result corresponding to the current audio frame is obtained.
[0014] Preferably, the process of obtaining the output audio data includes: Based on the feedback estimation results, main amplifier feedback estimation signals and monitor feedback estimation signals are generated for the main amplifier transmitted audio data and monitor transmitted audio data, respectively; the main amplifier feedback estimation signal and monitor feedback estimation signal are combined into a feedback estimation signal; the feedback estimation signal is subtracted from the corresponding microphone real-time audio data to obtain the output audio data.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By continuously associating candidate frequency components across channels and time frames in the real-time spectrum sequence of the microphone, the spectrum sequence of the main amplifier, and the spectrum sequence of the monitor, a closed-loop propagation relationship between the candidate frequency components and the microphone is established. This allows the determination of candidate frequency components to simultaneously combine their narrowband continuous characteristics and the cyclic propagation characteristics in the sound reinforcement system, providing closed-loop spectral data for subsequent source status identification and feedback path status updates of candidate frequency components.
[0016] 2. By performing continuous frame matching on the Mel frequency cepstral coefficients, changes in Mel frequency cepstral coefficients between adjacent frames, the duration of spectral peaks, the fundamental frequency and corresponding harmonics of the real-time audio frames corresponding to the candidate frequency components, the degree of correspondence between the candidate frequency components and the timbre trajectory during the audition stage can be characterized from two aspects: temporal continuity and timbre characteristic change relationship. This provides a basis for distinguishing the normal vocal components and feedback components of the actor.
[0017] 3. By generating feature masks from closed-loop spectral data, the timbre continuity sequence is subjected to restricted screening. Then, the restricted timbre features and closed-loop propagation features are fused using a gated cyclic network to determine the actor source state, feedback source state, and mixed source state of the candidate frequency components. The determination results are then converted into hold state, update state, and continue current state, thereby providing a targeted update control basis for the recursive derivation of Kalman feedback path states in different frequency bands. Attached Figure Description
[0018] Figure 1A schematic diagram of a method for intelligent suppression of howling based on audio feature recognition provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the data flow for forming a closed-loop spectral system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the process for obtaining the timbre continuity sequence provided in an embodiment of the present invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] Please see Figures 1 to 3 This invention provides a method for intelligent suppression of howling based on audio feature recognition, the technical solution of which is as follows: Example 1: A method for intelligent suppression of howling based on audio feature recognition includes: Acquire the main amplifier audio data, monitor audio data, test audio frame sequence, microphone real-time audio frame sequence, main amplifier audio frame sequence, and monitor audio frame sequence; extract Mel frequency cepstral coefficients and fundamental frequency sequence from the test audio frame sequence to form timbre trajectory data; Spectral peaks are extracted from the real-time microphone audio frame sequence to obtain candidate frequency component sequences. The cyclic relationship of the candidate frequency component sequences in the real-time microphone audio frame sequence, the main amplifier audio frame sequence, and the monitor audio frame sequence is traced to form closed-loop spectral data. Mel frequency cepstral coefficients and fundamental frequency sequences of the microphone real-time audio frames corresponding to the candidate frequency component sequences are extracted, and matched with timbre trajectory data to obtain timbre continuation sequences. The timbre continuation sequences and closed-loop spectrum data are input into a gated recurrent network to output a Kalman update sequence. The feedback estimation results are obtained by recursively deriving from the Kalman update sequence, using the main amplifier transmitted audio frame sequence and the monitor transmitted audio frame sequence as references and the microphone real-time audio frame sequence as observations. The feedback estimation signal is generated based on the feedback estimation result, the main amplifier audio data, and the monitor audio data; the feedback estimation signal is subtracted from the microphone real-time audio data to obtain the output audio data.
[0021] Furthermore, the process of acquiring the main speaker transmitted audio data, the monitor transmitted audio data, the test audio frame sequence, the microphone real-time audio frame sequence, the main speaker transmitted audio frame sequence, and the monitor transmitted audio frame sequence includes: Acquire microphone test audio data and microphone real-time audio data, as well as main amplifier transmit audio data and monitor transmit audio data; synchronize the microphone test audio data, microphone real-time audio data, main amplifier transmit audio data, and monitor transmit audio data with the same sampling frequency to obtain synchronized audio data; divide the synchronized audio data into frames with the same frame length and frame shift to obtain test audio frame sequence, microphone real-time audio frame sequence, main amplifier transmit audio frame sequence, and monitor transmit audio frame sequence, respectively.
[0022] In this embodiment, microphone test audio data, microphone real-time audio data, main amplifier transmit audio data, and monitor transmit audio data are first acquired.
[0023] Among them, the microphone test audio data is the actor's test signal collected through the same microphone input channel as the formal sound reinforcement stage before the formal sound reinforcement begins; the microphone real-time audio data is the audio signal continuously collected through the microphone input channel during the formal sound reinforcement stage; the main sound reinforcement transmission audio data is the digital audio signal output by the digital audio processing equipment to the corresponding transmission bus of the main sound reinforcement speaker; and the monitor transmission audio data is the digital audio signal output by the digital audio processing equipment to the corresponding transmission bus of the monitor speaker.
[0024] This embodiment uses a digital mixing console and a digital audio processor to construct the audio acquisition and processing equipment. The microphone input channel, main amplifier transmit bus, and monitor transmit bus all use the same audio processing clock. The microphone input signal is converted from analog to digital to form 24-bit pulse code modulation data. The main amplifier transmit audio data is read from the output node of the main amplifier transmit bus of the digital audio processor, and the monitor transmit audio data is read from the output node of the monitor transmit bus of the digital audio processor. The sampling frequency of the microphone test audio data, microphone real-time audio data, main amplifier transmit audio data, and monitor transmit audio data is all set to 48 kHz, and the pulse code modulation data of each channel is converted into floating-point audio data in the range of [-1, 1].
[0025] For microphone real-time audio data, main amplifier audio data, and monitor audio data, the digital audio processor sequentially numbers each sampling point according to a common audio processing clock, and associates the microphone real-time audio sample value, main amplifier audio sample value, and monitor audio sample value with the same sampling sequence number to the same time index to complete the time synchronization of the three channels of real-time audio data.
[0026] Both the main amplifier and monitor audio data are read from the output node of the corresponding transmit bus inside the digital audio processor. During synchronization, no compensation is made for the acoustic propagation delay between the main amplifier speaker and the microphone, or between the monitor speaker and the microphone. This ensures that the acoustic propagation delay is preserved in the time relationship between the microphone's real-time audio data and the corresponding transmitted audio data. However, the prior acoustic propagation delay of the main amplifier and monitor is extracted by calculating the generalized cross-correlation between the microphone and the transmitted signal during the sound check phase, which guides subsequent adaptive frame lookup.
[0027] For microphone test audio data, the first valid sampling point at the start of the test acquisition is used as the starting sampling point. Continuous sampling is performed at a sampling frequency of 48 kHz, and the data is stored using the same data format and sampling sequence numbering rules as the real-time microphone audio data. This yields microphone test audio data processed with unified sampling parameters, as well as time-synchronized real-time microphone audio data, main amplifier transmit audio data, and monitor transmit audio data.
[0028] Subsequently, the microphone test audio data processed with unified sampling parameters, the real-time microphone audio data with time synchronization, the main amplifier transmission audio data, and the monitor transmission audio data are framed with the same frame length and frame shift. In this embodiment, the frame length is set to 2048 sampling points, corresponding to 42.67 ms; the frame shift is set to 512 sampling points, corresponding to 10.67 ms; and adjacent audio frames overlap by 1536 sampling points.
[0029] For microphone test audio data, 2048 sampling points are continuously read from the starting sampling point to form the first test audio frame. Then, the starting point is moved forward by 512 sampling points to form the next test audio frame. This process is repeated to continuously divide the data into frames, resulting in a sequence of test audio frames arranged according to frame number. For data segments with fewer than 2048 sampling points at the end, no test audio frame is generated.
[0030] For microphone real-time audio data, main amplifier audio data, and monitor audio data, the same 2048-point frame length and 512-point frame shift are used for synchronous framing. Microphone real-time audio frames, main amplifier audio frames, and monitor audio frames with the same frame sequence number have the same start sampling sequence number and end sampling sequence number, thus obtaining microphone real-time audio frame sequence, main amplifier audio frame sequence, and monitor audio frame sequence, respectively.
[0031] This yields the test audio frame sequence, the microphone real-time audio frame sequence, the main amplifier transmit audio frame sequence, and the monitor transmit audio frame sequence. The test audio frame sequence is used to establish subsequent timbre trajectory data, while the microphone real-time audio frame sequence, the main amplifier transmit audio sequence, and the monitor transmit audio sequence are used for subsequent candidate frequency component extraction, closed-loop spectrum data formation, and feedback path state recursion.
[0032] Furthermore, the process of forming timbre trajectory data includes: Short-time Fourier transform and feature extraction are performed on the audio frame sequence of the test audio to obtain the spectral peak position, spectral peak duration, fundamental frequency sequence and corresponding harmonic sequence, Mel frequency cepstral coefficient and Mel frequency cepstral coefficient change sequence of each audio frame. According to the continuation order of the same fundamental frequency sequence and corresponding harmonic sequence in consecutive audio frames, the spectral peak position, spectral peak duration, fundamental frequency sequence and corresponding harmonic sequence, Mel frequency cepstral coefficient and Mel frequency cepstral coefficient change sequence of adjacent frames are continuously correlated to form timbre trajectory data.
[0033] In this embodiment, short-time Fourier transform and feature extraction are performed frame by frame on the aforementioned audio test frame sequence. Each audio test frame has a sampling frequency of 48 kHz, a frame length of 2048 sampling points, and a frame shift of 512 sampling points. After applying a 2048-point Hanning window to each audio test frame, a 2048-point fast Fourier transform is performed to obtain the amplitude spectrum with a frequency resolution of 23.4375 Hz.
[0034] In the amplitude spectrum of each test audio frame, the frequency points whose amplitudes are simultaneously greater than the adjacent frequency points and not less than twice the median of the current frame's amplitude spectrum are identified as spectral peaks, and their frequency positions are recorded. For adjacent test audio frames, when the frequency difference between two spectral peaks does not exceed 46.875 Hz, they are associated as continuous components of the same spectral peak, and the corresponding frame number, frequency position, and number of consecutive frames are recorded to form a continuous relationship of spectral peaks.
[0035] The fundamental frequency is extracted within the 80-1000 Hz range. The spectral peaks within this range are used as candidate fundamental frequencies. The 2nd to 5th harmonics are then searched sequentially. If at least three effective spectral peaks exist in the fundamental frequency and its 2nd to 5th harmonics, the candidate fundamental frequency is retained. The candidate fundamental frequency with the largest sum of the amplitudes of the corresponding effective spectral peaks is determined as the fundamental frequency of the current test audio frame. To overcome the discrete frequency resolution limitation of the Fast Fourier Transform, parabolic interpolation of the spectral peaks is performed on the determined fundamental frequency and effective harmonic peaks. The precise frequency position at the sub-frequency resolution level is calculated using the amplitude values of the peak frequency and its left and right adjacent frequencies. The corresponding precise harmonic frequencies are recorded according to the harmonic order to obtain the corresponding harmonic data. For adjacent test audio frames, if the absolute frequency difference of the fundamental frequency does not exceed one frequency resolution unit (23.4375 Hz), or the relative change ratio of the fundamental frequency does not exceed 6%, the corresponding fundamental frequency and its harmonics are continuously correlated to form a fundamental frequency sequence and its corresponding harmonic sequence.
[0036] Simultaneously, Mel-frequency cepstral coefficients are extracted based on the power spectrum of each audio test frame. Forty Mel-scale triangular filters covering 20-8000Hz are used to calculate the filtered frequency band energy. The natural logarithm of the filtered frequency band energy is taken, and a discrete cosine transform is performed, retaining the 13-dimensional Mel-frequency cepstral coefficients from order 0 to order 12. The 13-dimensional Mel-frequency cepstral coefficients of the subsequent audio test frame are subtracted from the corresponding coefficients of the previous audio test frame to obtain the Mel-frequency cepstral coefficient variation data between adjacent frames.
[0037] After completing the feature extraction, using the same fundamental frequency sequence and its corresponding harmonic sequence as continuous association indices, and following the temporal order of the test audio frames, the spectral peak positions, spectral peak durations, fundamental frequency and corresponding harmonics, Mel frequency cepstral coefficients, and Mel frequency cepstral coefficient changes in adjacent frames are continuously associated to form timbre trajectory data. Each timbre trajectory stores the above features sequentially according to its frame number, for use in subsequent continuous frame matching of candidate frequency components corresponding to real-time microphone audio frames.
[0038] Furthermore, the process of forming closed-loop spectral data includes: Short-time Fourier transforms are performed on the microphone real-time audio frame sequence, the main amplifier transmit audio frame sequence, and the monitor transmit audio frame sequence to obtain the microphone real-time spectrum sequence, the main amplifier transmit spectrum sequence, and the monitor transmit spectrum sequence, respectively. Narrowband spectral peaks that persist in continuous audio frames are extracted from the microphone real-time spectrum sequence to obtain a candidate frequency component sequence. For each candidate frequency component in the candidate frequency component sequence, transmit frequency components with the same frequency evolution order are searched in the main amplifier transmit spectrum sequence and the monitor transmit spectrum sequence, respectively. According to the time sequence of the transmit frequency component entering the microphone real-time spectrum sequence, re-entering the main amplifier transmit spectrum sequence and the monitor transmit spectrum sequence, and returning to the microphone real-time spectrum sequence again, and combined with the a priori sound propagation delay obtained during the sound test, the candidate frequency components are continuously correlated across frames to form closed-loop spectral data.
[0039] In this embodiment, a 2048-point Hanning window is applied to the aforementioned microphone real-time audio frame sequence, main amplifier transmit audio frame sequence, and monitor transmit audio frame sequence, and a 2048-point Fast Fourier Transform is performed to obtain the microphone real-time spectrum sequence, main amplifier transmit spectrum sequence, and monitor transmit spectrum sequence, with a frequency resolution of 23.4375 Hz.
[0040] Local spectral peaks are extracted frame by frame from the microphone's real-time spectrum sequence. When the same spectral peak exists in four consecutive audio frames, the frequency difference between peaks in adjacent frames does not exceed 46.875 Hz, and the bandwidth corresponding to a 3 dB drop on either side of the peak does not exceed 93.75 Hz, the peak is identified as a persistent narrowband peak. The frequency position and amplitude of these narrowband peaks are continuously correlated according to frame number to obtain a candidate frequency component sequence.
[0041] For each candidate frequency component, frequency trajectory matching is performed in both the main spread spectrum sequence and the listen-back spectrum sequence. Using the frequency position of the candidate frequency component in each consecutive frame as the matching benchmark, when the frequency difference between the spectral peak of the corresponding frame in the transmitted spectrum and the candidate frequency component does not exceed 46.875 Hz, and this condition is met for four consecutive frames, the corresponding spectral peak is determined as the transmitted frequency component with the same frequency evolution order.
[0042] The cyclic propagation relationship of the transmitted frequency components is then traced according to the time frame sequence. Starting from the frames containing the main spread spectrum frequency component and the monitor transmit frequency component, an adaptive search window is set based on the prior sound propagation delay: for the monitor transmit frequency component, the same frequency component in the microphone real-time spectrum is searched within 0-1 audio frames; for the main spread spectrum frequency component, the same frequency component is searched within 1-6 audio frames. For near-field monitor same frequency components falling within the same audio frame (i.e., 0 frame interval), the sub-frame delay is further calculated using the phase difference between the microphone real-time spectrum and the transmitted spectrum at the corresponding frequency points to determine the order of signal arrival. Within 1-2 audio frames after the appearance of the microphone same frequency component, the corresponding same frequency component is searched in the main spread spectrum sequence and the monitor transmit spectrum sequence; then, according to the corresponding transmission channel type, the same frequency component that reappears in the microphone real-time spectrum sequence is searched within the respective adaptive search windows. The frequency difference of each match is limited to no more than 46.875Hz.
[0043] When candidate frequency components are continuously associated in the time sequence of "transmitted frequency component, microphone real-time frequency component, retransmitted frequency component, and re-returned microphone real-time frequency component", the frame number, frequency position, amplitude, and the main amplifier transmission channel and monitor transmission channel to which each associated node belongs are recorded, forming a closed-loop propagation trajectory corresponding to that candidate frequency component. The closed-loop propagation trajectories corresponding to each candidate frequency component are stored continuously according to the frame number to form closed-loop spectrum data.
[0044] Furthermore, the process of obtaining the timbre continuity sequence includes: Mel frequency cepstral coefficients, Mel frequency cepstral coefficient variation sequences between adjacent frames, spectral peak duration relationships, fundamental frequency sequences, and corresponding harmonic sequences are extracted from the microphone real-time audio frame sequences corresponding to the candidate frequency component sequences. The Mel frequency cepstral coefficients, Mel frequency cepstral coefficient variation sequences between adjacent frames, spectral peak duration relationships, fundamental frequency sequences, and corresponding harmonic sequences are then continuously matched with the timbre trajectory data to obtain the timbre duration sequences corresponding to each candidate frequency component.
[0045] In this embodiment, the corresponding real-time microphone audio frame is determined based on the aforementioned candidate frequency component sequence, and the same feature extraction parameters as those used to form the timbre trajectory data are used to extract 13-dimensional Mel frequency cepstral coefficients, Mel frequency cepstral coefficient variation data of adjacent frames, spectrum peak duration relationship, fundamental frequency and corresponding 2nd to 5th order harmonics from the real-time microphone audio frame.
[0046] Based on the frame number of the candidate frequency components, eight consecutive real-time microphone audio frames are used as a matching window. The features in the matching window are compared frame by frame with the features in the eight consecutive frames of the timbre trajectory data. For Mel frequency cepstral coefficients and the changes in Mel frequency cepstral coefficients in adjacent frames, the Euclidean distance between the corresponding 13-dimensional feature vectors is calculated. For the fundamental frequency, the relative frequency difference between the real-time fundamental frequency and the fundamental frequency corresponding to the timbre trajectory is calculated. For the corresponding harmonics, the number of matching harmonics with a frequency difference not exceeding 46.875 Hz among the 2nd to 5th order harmonics is counted. For the duration of spectral peaks, the number of common spectral peaks that can maintain continuous correlation between the real-time matching window and the consecutive frames corresponding to the timbre trajectory is counted.
[0047] When the relative frequency difference of the fundamental frequency in eight consecutive frames does not exceed 6%, there are at least three matching harmonics in the corresponding harmonics of each frame, and the continuous spectral peaks maintain a corresponding relationship in at least six frames, the eight consecutive frames are determined to be continuously matched with the corresponding timbre trajectory. The Euclidean distance of the Mel frequency cepstral coefficients and the Euclidean distance of the Mel frequency cepstral coefficient changes in adjacent frames are set with matching thresholds based on the mean of the distances of the eight adjacent frames within the same timbre trajectory during the audition phase plus one standard deviation. At the same time, the real-time matching distance must also be less than the upper limit of the range between the minimum and maximum values of the distances of the eight adjacent frames within the timbre trajectory during the audition phase. When both conditions are met, it is considered that the timbre feature matching is satisfied.
[0048] For each candidate frequency component, the distance between Mel frequency cepstral coefficients in each matching window, the distance of Mel frequency cepstral coefficient changes between adjacent frames, the relative frequency difference of the fundamental frequency, the number of matched harmonics, the number of consecutive spectral peak matching frames, and the consecutive matching results are recorded sequentially according to the frame number, forming the timbre continuation sequence corresponding to that candidate frequency component. The timbre continuation sequence is stored continuously according to the time order of the microphone's real-time audio frames and serves as the input data for subsequent gated loop networks.
[0049] Furthermore, the process of outputting the Kalman update sequence includes: Feature masks for corresponding candidate frequency components are generated based on the closed-loop spectrum data; the feature masks are used to filter the timbre continuation sequence to obtain a restricted timbre sequence; the restricted timbre sequence and the closed-loop spectrum data are input into a gated recurrent network in time frame order, and the gated recurrent network outputs the actor source state, feedback source state, and mixed source state corresponding to each candidate frequency component; the candidate frequency components corresponding to the actor source state and their harmonic components with the same fundamental frequency relationship are marked as held states, the candidate frequency components corresponding to the feedback source state are marked as updated states, and the candidate frequency components corresponding to the mixed source state are marked as continued current states; the Kalman update sequence is generated according to the marked states and corresponding frequency positions of each candidate frequency component.
[0050] In this embodiment, based on the aforementioned closed-loop spectrum data, a feature mask is generated for each candidate frequency component according to the audio frame order. For the current candidate frequency component, when there is a closed-loop propagation trajectory in the closed-loop spectrum data with a frequency difference not exceeding 46.875Hz, and the continuous association of "the transmitted frequency component entering the microphone real-time spectrum, re-entering the main spread spectrum and the monitor transmission spectrum, and returning to the microphone real-time spectrum again" is completed, the mask value of the corresponding audio frame is set to 1; when the continuous association is not completed, the mask value is set to 0, and the feature mask corresponding to the candidate frequency component is formed by the mask values of the continuous audio frames.
[0051] The timbre continuity sequence is filtered frame by frame using the feature mask. When the mask value is 1, the following are retained in the corresponding audio frame: Mel frequency cepstral coefficient matching distance, Mel frequency cepstral coefficient change matching distance between adjacent frames, relative frequency difference of fundamental frequency, number of matched harmonics, number of consecutive spectral peak matching frames, and consecutive matching results. When the mask value is 0, the Mel frequency cepstral coefficient matching distance, Mel frequency cepstral coefficient change matching distance between adjacent frames, and relative frequency difference of fundamental frequency are set to 1, the number of matched harmonics and the number of consecutive spectral peak matching frames are set to 0, and the consecutive matching results are set to 0. These are used as placeholder values for restricted timbre features in the case of incomplete closed-loop association, and the restricted timbre sequence is obtained by arranging them according to the original frame number.
[0052] In this embodiment, each audio frame employs a 6-dimensional restricted timbre feature, and extracts candidate frequency position, spectral peak amplitude, main amplifier transmit matching marker, monitor transmit matching marker, frame interval between transmitted frequency components and microphone real-time frequency components, frame interval between returning to the microphone real-time spectrum, and a closed-loop completion marker from the closed-loop spectrogram data, forming a total of 13-dimensional input features. The closed-loop completion marker indicates whether the current candidate frequency component has completed the aforementioned closed-loop propagation association; when the closed-loop completion marker is 1, the corresponding 6-dimensional restricted timbre feature is the feature value after matching; when the closed-loop completion marker is 0, the corresponding 6-dimensional restricted timbre feature uses the aforementioned placeholder value. Therefore, the placeholder value is used in conjunction with the closed-loop completion marker and is not used alone to indicate whether the feature is valid. The spectral peak amplitude is normalized to the [0, 1] interval by the maximum amplitude of all candidate frequency components within the current audio frame; the frame interval between transmitting frequency components to the microphone's real-time frequency components and the frame interval between returning to the microphone's real-time spectrum are both normalized to the [0, 1] interval by dividing by 8 in units of frames; the main amplifier transmission matching flag, the listen-back transmission matching flag, and the loop closure completion flag are all set to 0 or 1. Sixteen consecutive audio frames constitute a gated cyclic network input sequence.
[0053] The gated recurrent network in this embodiment adopts a two-layer GRU structure. The first layer contains 64 hidden units, and the second layer contains 32 hidden units. The output of the second layer is connected to a fully connected layer with 3 output nodes. The classification probabilities corresponding to the actor source state, feedback source state, and mixed source state are output through the Softmax function. The state with the highest classification probability is determined as the source state of the current candidate frequency component.
[0054] The training data for the gated loop network was collected during the sound reinforcement system testing phase. Actor source training data was collected under the condition that the actor was speaking and the acoustic feedback paths of the main amplifier and monitor speakers were turned off; the extracted candidate frequency components were labeled as actor source states. Feedback source training data was collected under the condition that no actor was speaking and an acoustic feedback path was established through the main amplifier and monitor speakers; the extracted candidate frequency components were labeled as feedback source states. Mixed source training data was generated by digitally mixing the aforementioned actor source audio data and feedback source audio data at different signal ratios.
[0055] For candidate frequency components in the mixed-source training data, the energy proportion of the candidate frequency component at the corresponding frequency points of the pure actor source audio and the pure feedback source audio is calculated. When the energy proportion of the actor source component exceeds a preset upper limit threshold, the candidate frequency component is marked as an actor source state; when the energy proportion of the feedback source component exceeds the preset upper limit threshold, the candidate frequency component is marked as a feedback source state; when the energy proportions of both do not exceed the preset upper limit threshold, the candidate frequency component is marked as a mixed source state. Each candidate frequency component is assigned a unique source state label according to the above energy proportion rules. The closed-loop spectrum association result is used as the input feature of the gated recurrent network and is not used as the basis for determining the source state label.
[0056] Training samples were generated using the same feature extraction, closed-loop spectrum construction, and feature masking methods as in real-time processing. Each training sample contained 16 consecutive audio frames. Equal amounts of samples were drawn from actor source training data, feedback source training data, and mixed source training data, and then divided into training set, validation set, and test set in an 8:1:1 ratio.
[0057] For actor source training samples acquired under conditions of closed acoustic feedback path, the main amplification transmission matching flag, monitor transmission matching flag, and closed-loop completion flag in their closed-loop spectrum features are set to 0, while the frame interval for transmitting frequency components to the microphone's real-time frequency components and the frame interval for returning to the microphone's real-time spectrum are set to 8. Each training sample is labeled according to candidate frequency components: candidate frequency components in actor source training data are labeled as actor source states; candidate frequency components in feedback source training data are labeled as feedback source states; and candidate frequency components in mixed source training data are determined to be actor source states, feedback source states, or mixed source states according to the aforementioned energy proportion rules.
[0058] The closed-loop spectral association results serve as input features for the gated recurrent network. The loss function is jointly composed of classification cross-entropy loss and acoustic-physical constraint loss: ; in, The cross-entropy loss function is used for multi-class classification. The harmonic source consistency loss term is used to penalize the difference in source state output probabilities between the fundamental frequency candidate component and its 2nd to 5th harmonic components within the same audio frame. Its expression is: ; in, This represents the number of effective fundamental frequencies in the current training sequence. This represents the classification probability distribution vector corresponding to the i-th fundamental frequency component. This represents the classification probability distribution vector corresponding to its r-th harmonic component; This represents the physical constraint weighting coefficient (e.g., set to 0.2). The Adam algorithm is used based on the total loss. The parameters were updated, the learning rate was set to 0.001, the batch size was set to 64, and after training for 50 epochs, the network parameters with the lowest loss on the validation set were saved for real-time source state determination.
[0059] For the actor source state output by the gated recurrent network, the corresponding candidate frequency component is marked as a hold state. Based on the fundamental frequency of the audio frame containing the candidate frequency component and its corresponding harmonic sequence, the corresponding 2nd to 5th order harmonic components with a frequency difference not exceeding 46.875Hz are simultaneously marked as hold states. For the feedback source state, the corresponding candidate frequency component is marked as an update state. For the mixed source state, the corresponding candidate frequency component is marked as a continuation of the current state.
[0060] Finally, the frequency position of each candidate frequency component, its maintenance state, update state, and continuation state are recorded according to the audio frame number to form a Kalman update sequence, which serves as the control data for subsequent Kalman feedback path state recursion.
[0061] Furthermore, the process of recursively obtaining the feedback estimation result from the Kalman update sequence includes: The main amplifier transmit audio frame sequence and the monitor transmit audio frame sequence are used as reference audio data, and the microphone real-time audio frame sequence is used as the observed audio data. Kalman feedback path state recursion is performed band by band according to the Kalman update sequence. For frequency bands marked as "hold state", the feedback path state corresponding to the previous audio frame is maintained. For frequency bands marked as "update state", the feedback path state is updated according to the reference audio data and observed audio data corresponding to the current audio frame. For frequency bands marked as "continue current state", the feedback path state corresponding to the previous audio frame is used and the state determination is continued in the next audio frame. The feedback estimation result corresponding to the current audio frame is obtained.
[0062] In this embodiment, the effective frequency range of 0–24000Hz corresponding to the 2048-point Fast Fourier Transform is divided into K frequency bands, where K=1025, and each frequency band corresponds to one Fast Fourier Transform frequency point. The audio frames are then consecutively numbered according to their chronological order. Indicates the frame number of the current audio frame; This indicates the frequency band number currently being processed. =1, 2, ..., K. For each candidate frequency component, the frequency band with the smallest absolute value of its frequency difference is determined as the corresponding target frequency band, and a state maintenance, state update, or current state continuation flag is applied to the target frequency band; 2nd to 5th order harmonic components with the same fundamental frequency relationship as the candidate frequency components are mapped to their corresponding frequency bands and a synchronization flag is applied. For frequency bands not covered by any candidate frequency components, the feedback path state corresponding to the previous audio frame is maintained, and no feedback path state update is performed based on the reference audio data and observed audio data of the current audio frame. For the ... The first audio frame Each frequency band is composed of the main spread feedback path transfer coefficient and the echo feedback path transfer coefficient, forming the feedback path state vector: ; ; in, This represents the feedback path propagation coefficient from the main transmission channel to the microphone input channel corresponding to the m-th audio frame and the k-th frequency band. This represents the feedback path transfer coefficient from the monitor transmit channel to the microphone input channel corresponding to the m-th audio frame and the k-th frequency band. The main spread spectrum transmission coefficient and the monitor transmit spectrum transmission coefficient are recursively derived according to the state of consecutive audio frames. The feedback path state of the current audio frame is based on the feedback path state of the corresponding frequency band of the previous audio frame, rather than independently solving for the two feedback path transfer coefficients based on the observations of the current single audio frame. A reference vector is formed by combining the main spread spectrum value and the monitor transmit spectrum value of the m-th audio frame and the k-th frequency band. The corresponding microphone real-time spectrum value As an observation, the feedback observation relationship is expressed as: where, This represents the observation noise in the current frequency band. Before updating the state of the current audio frame, the feedback path state of the same frequency band in the previous audio frame is used as the prediction state of the current audio frame: ; The predicted state covariance is obtained based on the state covariance and process noise covariance of the previous audio frame: ; in, This represents the state of the prediction feedback path for the m-th audio frame and the k-th frequency band before the observation update. This represents the corresponding predicted state covariance matrix; Let represent the process noise covariance matrix of the k-th frequency band. For the frequency band marked as the update state, the Kalman gain is calculated according to the following formula: ; in, express The conjugate transpose of . Indicates the first The observation noise variance parameter for each frequency band. The following method is used to determine the frequency spectrum: Microphone audio data is collected for two consecutive seconds before the official sound reinforcement begins, during which no performer is speaking. This data is then subjected to the same frame-by-frame and short-time Fourier transform as in real-time processing to obtain the complex spectrum values for each frequency point in each frame; for the [missing information]... For each frequency band, take the real and imaginary parts of the complex spectrum values of all frames within that 2-second period at that frequency point, and calculate the variance of the real part and the variance of the imaginary part respectively. The sum of the real and imaginary variances is taken. Whether to use the current microphone's real-time spectrum observations to perform feedback path state updates during the formal sound reinforcement phase is determined by the aforementioned Kalman update sequence. The feedback path state is updated based on the observed residuals: ; After completing the feedback path state update, update the corresponding state covariance matrix according to the following formula: ; in, This represents the state covariance matrix after the observations of the m-th audio frame and the k-th frequency band have been updated. This represents the identity matrix corresponding to the dimension of the feedback path state vector. The updated main spread feedback path transfer coefficients and monitor feedback path transfer coefficients are used as the feedback path states for the current frequency band.
[0063] For frequency bands marked as "maintained," the feedback path state and state covariance corresponding to the previous audio frame are preserved, and Kalman updates are not performed based on the current reference audio data and observed audio data. For frequency bands marked as "continued current state," the feedback path state corresponding to the previous audio frame is used, and the process noise covariance is added to the state covariance of the previous audio frame to obtain the predicted state covariance corresponding to the current audio frame; simultaneously, the corresponding candidate frequency component is retained for the next audio frame, and the gated recurrent network continues to determine the source state by combining the data of the next audio frame. When the candidate frequency component is determined to be a feedback source state in a subsequent audio frame and is marked as an update state, the feedback path state update is performed using the reference audio data and observed audio data corresponding to that audio frame; while it is still determined to be a mixed source state, no feedback path state update based on the current observations is performed. In this embodiment, the diagonal elements of the process noise covariance matrix are set to 10. −5 .
[0064] After completing the state recursion of each frequency band of the current audio frame, the main spread feedback path transfer coefficient and the echo feedback path transfer coefficient corresponding to each frequency band are arranged according to the frequency position to obtain the feedback estimation result corresponding to the current audio frame.
[0065] Furthermore, the process of obtaining the output audio data includes: Based on the feedback estimation results, main amplifier feedback estimation signals and monitor feedback estimation signals are generated for the main amplifier transmitted audio data and monitor transmitted audio data, respectively; the main amplifier feedback estimation signal and monitor feedback estimation signal are combined into a feedback estimation signal; the feedback estimation signal is subtracted from the corresponding microphone real-time audio data to obtain the output audio data.
[0066] In this embodiment, based on the main spread feedback path transfer coefficient and the echo feedback path transfer coefficient corresponding to each frequency band in the aforementioned feedback estimation results, the feedback signals of the main spread transmitted audio data and the echo transmitted audio data are reconstructed respectively.
[0067] For the The first audio frame For each frequency band, the main spread transmission spectrum value is multiplied by the corresponding main spread feedback path transfer coefficient, and the listen-back transmission spectrum value is multiplied by the corresponding listen-back feedback path transfer coefficient to obtain the main spread feedback estimated spectrum and the listen-back feedback estimated spectrum, respectively. ; ; The main spread feedback estimation spectrum and the listen-back feedback estimation spectrum corresponding to the same audio frame and frequency band are added together to obtain the feedback estimation spectrum. The feedback estimation spectra of each frequency band are arranged according to their frequency positions, and after completing the conjugate symmetric spectrum, a 2048-point inverse fast Fourier transform is performed. The spectra are then overlapped and added according to the frame shift of the aforementioned 512 sampling points to obtain the feedback estimation signal.
[0068] The feedback estimation signal follows the sampling sequence relationship of the corresponding audio frames. Microphone real-time audio frames, main amplifier transmit audio frames, and monitor transmit audio frames with the same frame number have the same start and end sampling sequence numbers. The microphone real-time audio data maintains a correspondence according to the sampling sequence number. After completing feature extraction, source state determination, feedback path state recursion, and feedback estimation signal generation for the audio frame, the generated feedback estimation signal is mapped to the microphone real-time audio data of its corresponding audio frame according to the original sampling sequence number.
[0069] The primary feedback estimation spectrum is transformed by inverse fast Fourier transform and then overlapped and added to form the primary feedback estimation signal. The monitor feedback estimation spectrum is processed in the same way to form the monitor feedback estimation signal. The two signals are then added point-by-point according to the same sampling number to form the feedback estimation signal. After the corresponding feedback estimation signal is generated, it is subtracted point-by-point from the real-time microphone audio data with the corresponding sampling number to obtain the output audio data. ; in, This indicates the microphone's real-time audio data. This represents the feedback estimation signal synthesized from the main amplification feedback estimation signal and the echo feedback estimation signal. This indicates the output audio data. It is output sequentially according to the sample number. The output audio data is obtained.
[0070] Example 2 A method for intelligent suppression of howling based on audio feature recognition, wherein the process of forming timbre trajectory data by continuous association includes: acquiring a sequence of audition audio frames covering multiple registers and multiple dynamic ranges, and extracting the fundamental frequency and corresponding harmonic components; in the continuous audition audio frame sequence, tracking frequency components with the same fundamental frequency value and its integer multiples, and establishing the continuity order of the same fundamental frequency and corresponding harmonic family in continuous frames; according to the continuity order, performing frame-level synchronization alignment of the spectral peak position, spectral peak duration relationship, fundamental frequency sequence and corresponding harmonic sequence, Mel frequency cepstral coefficient and the Mel frequency cepstral coefficient change sequence of adjacent frames, and establishing a multidimensional feature mapping set corresponding to different registers and dynamic ranges; arranging the multidimensional feature mapping set into a multidimensional feature state evolution sequence according to the time frame sequence, and using the multidimensional feature state evolution sequence as timbre trajectory data.
[0071] This embodiment further explains the formation process of timbre trajectory data based on Embodiment 1. Except for the following, the audio acquisition, time synchronization, framing, short-time Fourier transform, and feature extraction processes are the same as in Embodiment 1.
[0072] Before the formal sound reinforcement begins, the actors undergo multi-range, multi-velocity step-by-step sound checks. First, the lowest and highest effective fundamental frequencies (FFCs) are determined based on the actors' continuous pitch rises and falls. The range from the FFC to the highest effective fundamental frequency is then divided into low, mid, and high ranges. The actors continuously produce sound in each range at three different vocdivo (weak, medium, and strong) velocities, with each range and vocdivo combination having a continuous sound duration of 4 seconds. The root mean square (RMS) levels of the microphone sound check audio data corresponding to the weak, medium, and strong vocdivo velocities are -30 to -24 dB, -21 to -15 dB, and -12 to -6 dB, respectively. The data is processed using the framing method described in Example 1 to obtain a sequence of sound check audio frames covering different ranges and vocdivo conditions. The timbre trajectory data established in this example corresponds to the actor who has completed the sound check and is used in the formal sound reinforcement stage following the sound check, using the same microphone input channel.
[0073] The fundamental frequency and corresponding harmonic components are extracted frame by frame from the audio test frame sequence. For adjacent audio test frames, the change in the fundamental frequency of the later frame relative to the fundamental frequency of the previous frame does not exceed a certain percentage. At that time, the corresponding fundamental frequencies of two frames are determined to be a continuous continuation of the same fundamental frequency. The fundamental frequency of the current frame is used as the basis for this determination. Based on, for Step to Harmonic matching is performed at integer multiples of the frequency position, when the corresponding spectral peak and The frequency difference between them does not exceed At that time, the spectral peak was determined to be the first First harmonic, of which The above matching is performed continuously according to the frame number to establish the continuity order of the same fundamental frequency and corresponding harmonic family in consecutive test audio frames.
[0074] According to the aforementioned continuation order, using the audio frame number of the audition as a unified index, the spectral peak position, spectral peak duration, fundamental frequency, corresponding harmonic sequence, Mel frequency cepstral coefficient, and Mel frequency cepstral coefficient change data of the same audition audio frame are synchronized at the frame level. Corresponding register and velocity identifiers are added to each frame to form frame-level multidimensional feature data.
[0075] A set of consecutive frame-level multidimensional feature data with the same register and velocity identifiers is created to establish a multidimensional feature mapping set under the corresponding register and velocity conditions. Then, the frame-level multidimensional feature data in the multidimensional feature mapping set are arranged in ascending order of frame number to form a multidimensional feature state evolution sequence.
[0076] The multi-dimensional feature state evolution sequences formed in the low, mid, and high frequency ranges under three dynamic conditions (weak, medium, and strong) are stored separately and collectively constitute the timbre trajectory data. Subsequently, when performing continuous timbre matching on candidate frequency components, the corresponding frequency range and dynamic condition are determined based on the fundamental frequency range and root mean square level of the microphone's real-time audio frame containing the candidate frequency component, and the corresponding multi-dimensional feature state evolution sequence is invoked for continuous frame matching.
[0077] A method for intelligent suppression of howling based on audio feature recognition, wherein the process of establishing a multi-dimensional feature mapping set corresponding to different pitch ranges and dynamic conditions includes: within the same time frame, comparing the frequency position of the spectral peak with the fundamental frequency sequence and the corresponding harmonic sequence, and extracting the effective harmonic peak positions that coincide with the fundamental frequency sequence and the corresponding harmonic sequence; constructing a harmonic structure matrix by constructing the effective harmonic peak positions and the duration relationship of the spectral peaks in consecutive time frames according to the continuity order; constructing a timbre envelope matrix by constructing the Mel frequency cepstral coefficients in consecutive time frames and the Mel frequency cepstral coefficient change sequence in adjacent frames according to the continuity order; and aligning the harmonic structure matrix and the timbre envelope matrix corresponding to the multi-pitch range and multi-dynamic step-by-step sound test conditions with time frames and concatenating features to generate a multi-dimensional feature mapping set.
[0078] In this embodiment, when forming a multi-dimensional feature mapping set under the aforementioned different pitch ranges and dynamic conditions, the spectral peak position within each time frame is first compared with the fundamental frequency and 2nd to 5th harmonic frequencies corresponding to that frame, according to the audio frame sequence number. Let the fundamental frequency of the current audio frame be... The target frequency of order r is ,in Find the spectral peak with the smallest distance to the r-th order target frequency in the current frame's spectral peak positions, when the condition is met. At that time, the peak value of the spectrum The effective harmonic peak position of the corresponding order is determined. If the above conditions are not met, the effective harmonic peak position of that order in the current frame is recorded as 0.
[0079] Based on the continuity order of the fundamental frequency and its corresponding harmonic family in consecutive audio test frames, the audio test frames under the same register and dynamic conditions are sorted. For each audio test frame, the positions of the effective harmonic peaks from the 1st to the 5th order and the number of consecutive frames in which each effective harmonic peak appears are recorded. The above data of M consecutive audio test frames are arranged by frame number to obtain a harmonic structure matrix. This harmonic structure matrix includes 10 columns, where columns 1 to 5 represent the positions of the effective harmonic peaks from the 1st to the 5th order, and columns 6 to 10 represent the number of consecutive frames in which the corresponding effective harmonic peaks exist.
[0080] The 13-dimensional Mel-frequency cepstral coefficients of each audition audio frame and the 13-dimensional changes in Mel-frequency cepstral coefficients between adjacent frames are arranged according to the same frame number, resulting in an M-row, 26-column timbre envelope matrix. Both the harmonic structure matrix and the timbre envelope matrix use the audition audio frame number as the row index, ensuring that data recorded in the same row corresponds to the same audition audio frame.
[0081] After aligning the frame numbers, the harmonic structure matrix and the timbre envelope matrix are concatenated along the feature dimension to obtain an M-row, 36-column feature matrix. The feature matrices obtained for the low, middle, and high registers under weak, medium, and strong dynamic conditions are assigned corresponding register and dynamic labels, and stored according to these labels to obtain multi-dimensional feature mapping sets corresponding to different registers and dynamic conditions.
[0082] A method for intelligently suppressing howling based on audio feature recognition, wherein the process of extracting the effective harmonic peak positions that coincide with the fundamental frequency sequence and the corresponding harmonic sequence includes: extracting the frequency center values of each frequency component in the fundamental frequency sequence and the corresponding harmonic sequence; establishing a frequency tolerance interval corresponding to each frequency component based on the frequency center values and a preset frequency offset parameter; determining whether each spectral peak position within the same time frame is located within the frequency tolerance interval; extracting the spectral peak positions located within the frequency tolerance interval as effective harmonic peak positions, and filtering out the spectral peak positions located outside the frequency tolerance interval.
[0083] In this embodiment, the aforementioned fundamental frequency sequence and corresponding harmonic sequence are processed frame by frame according to the test audio frame number. For the first... For each test audio frame, the fundamental frequency value of the frame is used as the frequency center value of the first-order frequency component, and the harmonic frequencies recorded in the corresponding second to fifth-order harmonic sequences are used as the frequency center values of the corresponding order frequency components.
[0084] A corresponding frequency tolerance range is established based on the values of each frequency center. The frequency resolution of the aforementioned 2048-point Fast Fourier Transform is... In this embodiment, the preset frequency offset parameter is set to two frequency resolution units, namely For the first The frequency components of order one, whose frequency center values are denoted as . The corresponding frequency tolerance range is set as follows: ; The positions of each spectral peak extracted within the same audio frame are compared with the aforementioned frequency tolerance range. When the spectral peak position... satisfy When, the position of the spectral peak is determined as the first... The effective harmonic peak position corresponding to the first frequency component; when the spectral peak position is outside the entire frequency tolerance range, the spectral peak is filtered out from the harmonic structure features of the current frame.
[0085] When there are more than two spectral peaks within the same frequency tolerance range, calculate the frequency difference between each spectral peak and its corresponding frequency center value, and retain the spectral peak with the smallest absolute value of the frequency difference as the effective harmonic peak position of that order. Perform the above processing sequentially according to the 1st to 5th order frequency components to obtain the effective harmonic peak positions corresponding to the current test audio frame, which are then used to construct the harmonic structure matrix.
[0086] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for intelligent suppression of howling based on audio feature recognition, characterized in that, include: Acquire the main amplifier audio data, monitor audio data, test audio frame sequence, microphone real-time audio frame sequence, main amplifier audio frame sequence, and monitor audio frame sequence; extract Mel frequency cepstral coefficients and fundamental frequency sequence from the test audio frame sequence to form timbre trajectory data; Spectral peaks are extracted from the real-time microphone audio frame sequence to obtain candidate frequency component sequences. The cyclic relationship of the candidate frequency component sequences in the real-time microphone audio frame sequence, the main amplifier audio frame sequence, and the monitor audio frame sequence is traced to form closed-loop spectral data. Mel frequency cepstral coefficients and fundamental frequency sequences of the microphone real-time audio frames corresponding to the candidate frequency component sequences are extracted, and matched with timbre trajectory data to obtain timbre continuation sequences. The timbre continuation sequences and closed-loop spectrum data are input into a gated recurrent network to output a Kalman update sequence. The feedback estimation results are obtained by recursively deriving from the Kalman update sequence, using the main amplifier transmitted audio frame sequence and the monitor transmitted audio frame sequence as references and the microphone real-time audio frame sequence as observations. A feedback estimation signal is generated based on the feedback estimation results, the main amplifier transmitted audio data, and the monitor transmitted audio data. The output audio data is obtained by subtracting the feedback estimation signal from the real-time microphone audio data.
2. The intelligent howling suppression method based on audio feature recognition according to claim 1, characterized in that, The process of acquiring the main amplifier audio data, monitor audio data, test audio frame sequence, microphone real-time audio frame sequence, main amplifier audio frame sequence, and monitor audio frame sequence includes: acquiring microphone test audio data and microphone real-time audio data, as well as main amplifier audio data and monitor audio data; synchronizing the microphone test audio data, microphone real-time audio data, main amplifier audio data, and monitor audio data with the same sampling frequency to obtain synchronized audio data; and dividing the synchronized audio data into frames with the same frame length and frame shift to obtain the test audio frame sequence, microphone real-time audio frame sequence, main amplifier audio frame sequence, and monitor audio frame sequence, respectively.
3. The intelligent howling suppression method based on audio feature recognition according to claim 1, characterized in that, The process of forming timbre trajectory data includes: performing short-time Fourier transform and feature extraction on the audio frame sequence of the test audio to obtain the spectral peak position, spectral peak duration relationship, fundamental frequency sequence and corresponding harmonic sequence, Mel frequency cepstral coefficient and Mel frequency cepstral coefficient change sequence of each audio frame; and continuously associating the spectral peak position, spectral peak duration relationship, fundamental frequency sequence and corresponding harmonic sequence, Mel frequency cepstral coefficient and Mel frequency cepstral coefficient change sequence of adjacent frames according to the continuity order of the same fundamental frequency sequence and corresponding harmonic sequence in consecutive audio frames to form timbre trajectory data.
4. The intelligent howling suppression method based on audio feature recognition according to claim 1, characterized in that, The process of forming closed-loop spectrum data includes: performing short-time Fourier transforms on the microphone real-time audio frame sequence, the main amplifier transmission audio frame sequence, and the monitor transmission audio frame sequence respectively to obtain the microphone real-time spectrum sequence, the main amplifier transmission spectrum sequence, and the monitor transmission spectrum sequence; extracting narrowband spectral peaks that persist in continuous audio frames from the microphone real-time spectrum sequence to obtain a candidate frequency component sequence; for each candidate frequency component in the candidate frequency component sequence, searching for transmission frequency components with the same frequency evolution order in the main amplifier transmission spectrum sequence and the monitor transmission spectrum sequence respectively; and performing cross-frame continuous association of candidate frequency components according to the time sequence of transmission frequency components entering the microphone real-time spectrum sequence, re-entering the main amplifier transmission spectrum sequence and the monitor transmission spectrum sequence, and returning to the microphone real-time spectrum sequence again to form closed-loop spectrum data.
5. The intelligent howling suppression method based on audio feature recognition according to claim 1, characterized in that, The process of obtaining the timbre continuity sequence includes: extracting Mel frequency cepstral coefficients, Mel frequency cepstral coefficient change sequences of adjacent frames, spectral peak duration relationships, fundamental frequency sequences, and corresponding harmonic sequences from the microphone real-time audio frame sequences corresponding to the candidate frequency component sequences; performing continuous frame matching of the Mel frequency cepstral coefficients, Mel frequency cepstral coefficient change sequences of adjacent frames, spectral peak duration relationships, fundamental frequency sequences, and corresponding harmonic sequences with the timbre trajectory data to obtain the timbre continuity sequence corresponding to each candidate frequency component.
6. The intelligent howling suppression method based on audio feature recognition according to claim 1, characterized in that, The process of outputting the Kalman update sequence includes: generating feature masks for corresponding candidate frequency components based on the closed-loop spectrum data; filtering the timbre continuation sequence using the feature masks to obtain a restricted timbre sequence; inputting the restricted timbre sequence and the closed-loop spectrum data into a gated recurrent network in time frame order, and having the gated recurrent network output the actor source state, feedback source state, and mixed source state corresponding to each candidate frequency component; marking the candidate frequency components corresponding to the actor source state and their harmonic components with the same fundamental frequency relationship as the hold state, marking the candidate frequency components corresponding to the feedback source state as the update state, and marking the candidate frequency components corresponding to the mixed source state as the continuation of the current state; and generating the Kalman update sequence according to the marked state and corresponding frequency position of each candidate frequency component.
7. The intelligent howling suppression method based on audio feature recognition according to claim 1, characterized in that, The process of obtaining the feedback estimation result by recursion based on the Kalman update sequence includes: using the main amplifier transmitted audio frame sequence and the monitor transmitted audio frame sequence as reference audio data, and using the microphone real-time audio frame sequence as observed audio data; performing Kalman feedback path state recursion band by band according to the Kalman update sequence; maintaining the feedback path state corresponding to the previous audio frame for frequency bands marked as "hold state", updating the feedback path state for frequency bands marked as "update state" based on the reference audio data and observed audio data corresponding to the current audio frame, and using the feedback path state corresponding to the previous audio frame for frequency bands marked as "continue current state" and entering the next audio frame to continue state determination; and obtaining the feedback estimation result corresponding to the current audio frame.
8. The intelligent howling suppression method based on audio feature recognition according to claim 1, characterized in that, The process of obtaining the output audio data includes: generating a main amplifier feedback estimation signal and a monitor feedback estimation signal for the main amplifier transmitted audio data and the monitor transmitted audio data respectively based on the feedback estimation results; combining the main amplifier feedback estimation signal and the monitor feedback estimation signal into a feedback estimation signal; and subtracting the feedback estimation signal from the corresponding microphone real-time audio data to obtain the output audio data.
Citation Information
Patent Citations
Audio signal processing method, model training method and related device
CN111210021A
Channel normalization apparatus and method for robustspeech recognition
KR1020060067097A
